Research whitepaper · November 2026
A usable memory of human activity
Research proposal · Edition 2 · November 2026
MathaLabs is building a way to turn first-person video into a structured record of what happened, when it happened and what evidence supports it. The intended result is a personal activity transcript that calendars, applications and AI assistants can use with the wearer’s permission. Our first research objective is to recover at least 90% of useful, observable activity events on a representative wearable-video benchmark while maintaining at least 95% precision. Both numbers are proposed targets. This paper defines the task, the limits of camera observation, the data and model strategy, and the experiments needed to test the claim. It starts with recorded clips and progresses toward an incremental system on a companion phone. The first three weeks are designed to produce a trustworthy baseline and an informed next investment in data and compute.
1. Give assistants a memory they can check
A person can ask an assistant to find a document because the document has been saved. Much of daily life has no equivalent record. Where did I leave the charger? Did I put the food away? Which assembly step did I finish? A wearable camera can capture evidence for these questions, but hours of video are difficult to search and expensive to process repeatedly.
The product we want is a compact, timestamped account of meaningful activity. A useful record links an action to its objects, its time and the footage that supports it. An assistant could retrieve the relevant interval, explain its answer and offer a reminder or calendar draft. The wearer decides which applications receive that context and which actions proceed.
Ego4D’s episodic-memory benchmark already asks systems to locate visual evidence for questions about past experience. EgoLife studies daily-life questions across long recordings and combines clip understanding with memory retrieval. These establish relevant research tasks. Our proposal extends that direction into a reusable event interface, evaluated on the cameras and activities our users actually bring.
- Capture a useful activity with permission and preserve its original time and source.
- Turn the evidence into a searchable event record, with uncertainty and later corrections.
- Let an authorized assistant retrieve that record, inspect its evidence and help the user complete a task.
2. What exists, and what we are building next
HumanCapture provides the software foundation for recording, importing, categorizing and uploading first-person clips. HumanHUD explores the wearable interface. Our research work has mapped major external POV sources and the supervision they provide. The native capture path currently requests a medium-resolution, 24-frame-per-second glasses stream and records video-only clips.
External sources are cataloged, and accepted training footage is still being assembled. Publisher-reported footage remains external supply until access, permitted use, quality and duplicate checks are complete. Our next deliverable is a permissioned training corpus and a measured video-to-event baseline. We have not established an end-to-end model accuracy result.
The initial product input is recorded video. We will evaluate audio as a separate extension when synchronized audio is available and authorized. Meta’s wearable toolkit connects glasses to a companion application, so the first deployment investigation is a phone-based pipeline with a measured choice between local and cloud inference.
3. Make the 90% objective testable
“Capture 90% of useful human activity” needs a defined denominator. We propose event recall as the primary measure. Human reviewers identify useful events that are observable in the recorded input. A recovered event must describe the correct action and relevant objects and match the reference interval under published timing rules. The target is at least 90% recall while precision remains at least 95%. The threshold is chosen using validation data, then held fixed for the final test.
Recall is matched reference events divided by all eligible reference events. Precision is matched predictions divided by all accepted predictions. Matching is one-to-one, so repeating an event cannot increase recall. An abstention on an observable reference event counts as a miss. A confident event without matching evidence counts as a false positive. A readable JSON object alone earns no semantic credit.
For example, a test with 100 eligible events, 90 correct matched predictions and 4 incorrect predictions has 90% recall and about 95.7% precision. It clears the proposed point-estimate targets. A defensible reported result also needs uncertainty intervals and a representative sample. We will publish session-grouped confidence intervals, category results and the recording conditions alongside the aggregate.
Coverage has its own report. We will measure which planned activities, people, environments and capture conditions were tested, how much consented recording time is usable, and how much activity remains outside the observable task. A 90% result in kitchens cannot support a claim about 90% of daily life.
| Measure | Proposed rule | What it prevents |
|---|---|---|
| Useful-event recall | At least 90% of eligible reference events recovered | A system that emits only easy events |
| Accepted-event precision | At least 95% at the same operating threshold | A system that fills a timeline with plausible guesses |
| Temporal grounding | Match action and objects; interval IoU at least 0.5 for duration events; point events within ±2 seconds | Correct words attached to the wrong moment |
| Coverage | Report activity, wearer, environment, observability and usable-time coverage separately | A narrow benchmark presented as universal ability |
4. Define useful and observable activity
We define a useful event as an observable action or state change that a person could reasonably want to retrieve or use in a task. Putting a charger in a drawer, starting a workout and completing an assembly step qualify. Every video frame and every repeated hand motion do not need a separate event. The annotation guide will specify event granularity before model results are inspected.
The initial taxonomy spans food preparation, household work, exercise, work and study, travel, shopping, outdoor activity, hobbies and social activity. Each domain includes background time, interruptions, unsuccessful actions, occlusion and transitions. Repeated actions can form one interval, while a consequential state change receives its own event. Coarse scenarios and fine actions are scored separately.
Observability is determined from the available input. A hand may be hidden, a label unreadable or a recording interrupted. Reviewers mark these intervals and the missing evidence. They can use a second view or the wearer’s notes to discover that an event occurred, but an event enters the visual-recall denominator only if the target video supports the required answer. Excluded events and reasons remain in the coverage report.
This scope keeps the objective ambitious and honest. We want broad recovery of useful events that the camera can support, alongside an explicit account of the life it misses. Measuring full-day usefulness will eventually require longer sessions and participant review beyond the initial short-clip pilot.
5. Store observations, interpretations and evidence separately
The event record is the contract between perception and assistance. “Placed a mug on the counter” can be a visual observation. “Preparing for a meeting” is an interpretation of a broader situation. The model should preserve both when useful, with separate evidence and uncertainty. A scenario must never silently become proof that every expected action occurred.
A record carries a source recording, a time interval, the action, relevant objects, evidence references, model version and status. An observation cutoff states the latest input available when it was produced. A stable event identifier lets later outputs revise an ongoing event instead of creating duplicate facts. Corrections remain traceable.
In the illustrative record below, the observation is that a charger was placed in a drawer. Packing for travel is a separate, uncertain interpretation. The timestamps are relative to a fictional source recording. An assistant answering where the charger was last seen should retrieve the observed event, its time and its supporting clip.
User edits receive their own provenance. They can correct the personal record immediately. Reusing those edits for model training requires the user’s training permission and a separate quality review.
| Record field | Meaning |
|---|---|
| event_id and revision | Stable identity and correction history |
| recording_id, start_s, end_s | Location on the original recording clock |
| action and objects | Directly supported event content |
| scenario and inference_status | Higher-level interpretation with separate uncertainty |
| evidence_refs and observation_cutoff | Frames or intervals used, and the latest information available |
| status, model_version, source | Ongoing/completed/uncertain state and reproducible provenance |
{
"event_id": "example-event-17",
"revision": 2,
"source": {
"recording_id": "example-recording",
"start_s": 192.0,
"end_s": 196.0
},
"observed": {
"action": "place",
"object": "charger",
"destination": "desk drawer"
},
"inferred": {
"scenario": "packing for travel",
"status": "uncertain"
},
"status": "completed",
"evidence": [
{
"recording_id": "example-recording",
"start_s": 192.0,
"end_s": 196.0
}
],
"observation_cutoff_s": 200.0,
"model_version": "illustrative-baseline-v0",
"review_status": "unreviewed"
}6. Choose data for the supervision it provides
The first corpus should teach the event contract and expose the model’s weaknesses. Its value depends on visible actions, reliable intervals, relevant settings and permitted use. An external collection’s size does not tell us how many distinct, useful training events it contributes.
RekaDaily’s processed release is a candidate for broad clip-language adaptation. Its publisher identifies captions and question-answer pairs as model-generated and unverified, while session activities come from reviewed labels. These require sample audits. Ego4D Goal-Step supplies nested goals, steps and substeps. EPIC-KITCHENS supplies fine action intervals and verb-noun labels. Ego-Exo4D’s atomic labels identify whether an action is visible to the wearer, a useful filter when annotation used other camera views.
EgoLife is a research reference for long-term memory, while our own consented glasses recordings establish target-camera validity. Large industrial or manipulation collections become later additions when error analysis identifies a relevant gap. Camera placement, field of view, motion blur, lighting and task mix all change the input distribution.
Every included sample needs an access and permitted-use record for its exact release. Code licenses, annotation licenses and video terms can differ. Eligibility for internal training, transformed data and redistribution must each be checked. The training manifest includes only cleared samples. Published source hours stay separate from acquired hours, deduplicated hours and accepted labeled hours.
| Source family | Useful supervision | Main experiment or limitation |
|---|---|---|
| RekaDaily processed | Clip captions, QA and reviewed session activities | Audit generated labels and reconstruct clip-to-parent time offsets |
| Ego4D Goal-Step | Goal, step and substep intervals | Learn hierarchy without turning goals into observed actions |
| EPIC-KITCHENS-100 | Action intervals and verb-noun classes | Learn hand-object boundaries; measure kitchen-to-other-domain transfer |
| Ego-Exo4D | Procedural steps and atomic descriptions | Select ego-visible wearer actions; keep synchronized views together |
| EgoLife | Long-session captions and memory questions | Study retrieval across time; group shared sessions across wearers |
| Our consented glasses video | Human-reviewed target-camera events | Measure real transfer and collect specific model failures |
7. Improve recall through measured adaptation
We will start with a fixed pretrained video-language baseline. The proposed first candidate is Qwen3-VL-4B-Instruct, with 2B for a lower-cost comparison and 8B as a reference. Qwen3-VL provides timestamp-text alignment and official fine-tuning code. Those capabilities make it a practical starting point for our experiments.
The first ablations compare 1, 2 and 4 sampled frames per second, short and overlapping longer windows, and a fixed visual-token budget. Higher sampling can recover brief actions, while longer context can resolve an object’s role. At a fixed token budget, those choices trade against per-frame detail and can change runtime. We will inspect missed events before deciding whether they need more frames, better temporal context, different examples or a better model.
Adaptation then uses reviewed clip-to-event examples, including idle intervals, ambiguous objects, interrupted actions and near misses. A point narration remains point supervision unless a reviewer supplies its boundaries. Missing labels are masked as unknown rather than treated as proof that nothing happened. A collector’s chosen category is weak metadata until verified against the video.
The first training experiment uses low-rank adaptation of attention projections with the vision backbone frozen. We will begin with rank 8, then change capacity only if validation failures justify it. The target is reviewed event JSON with original-source timestamps. The 2B and 4B runs test whether adaptation improves recall at the precision floor. One selected 8B adaptation follows only if its reference performance and measured cost justify the run.
Raw recordings, cropped clips, re-encodings and alternate camera views must share a lineage identifier. Exact hashes catch identical files; frame-based similarity and metadata find likely transformed duplicates. All descendants of a source session stay in the same split. Participant, environment and time separation test transfer beyond memorized scenes.
AI can propose labels, find inconsistencies and rank failures for review. Reviewers set the truth for validation and test data. A teacher-generated caption is a training candidate, and a second model agreeing with it is evidence to inspect. We will compare prompt-only, external-data adaptation and external-plus-target-data adaptation using the same held-out protocol.
8. Build a pipeline that preserves time and causality
The recorded-clip pipeline has five stages. Ingestion validates media and stores original timestamps. Sampling selects frames under a fixed budget. Perception proposes structured events. A verifier checks the schema, timestamps and evidence. A session reconciler merges overlapping predictions and writes revisions to the event store. The original media clock remains authoritative through every transcode and clip.
The incremental version reuses this pipeline but imposes a causal cutoff. At time t, inference sees only frames and memory available through t. It can publish an ongoing event and later close or revise it. A bounded queue, ordered sequence numbers and cancellation rules prevent stale results from overwriting newer state. Offline and causal replay receive separate scores because future frames can make an ambiguous earlier event easier to recognize.
For long-term memory, retrieval first narrows a time range, then returns specific events and source intervals. Summaries help locate evidence but retain links to the underlying records. EgoLife motivates this hierarchy. EgoProbe tests a harder case in which similar intervening experiences interfere with the exact target detail. We will include repeated visits and repeated object movements, scoring the retrieved occurrence and its evidence rather than accepting a plausible general answer.
The first deployment target is the companion phone, with cloud inference available for research. After establishing a reliable teacher baseline, we will compare a smaller student model, distillation and 8-bit or 4-bit weight quantization. The same event contract lets us change the inference engine without replacing the transcript or agent interface. Bounded visual tokens, cached recent context and event-triggered updates are candidate ways to reduce work. Each must preserve recall and the precision floor under causal replay.
We will measure sustained memory, battery, heat, bandwidth and response latency on the actual device, including the median and 95th-percentile delay from an observable event to a published update. Tests must run long enough to reveal thermal throttling and include network loss. Video text and spoken content are untrusted observations. They do not gain authority to change permissions or instruct another assistant merely by entering the event record.
9. Test omissions, hallucinations and consequential errors
The initial pilot follows the existing plan for 300 human-reviewed clips, with 100 for validation and 200 held out for testing. Session groups must be assigned before frames or clips are split. A second stress set uses continuous sessions containing background activity and transitions. Sampling only interesting clips would hide false alarms and failures to decide when an event begins.
Two reviewers independently label an overlap sample and adjudicate disagreements. The guide records uncertainty, alternate valid names and boundary tolerances. We will publish agreement rates and failure examples. Initial timing rules use interval intersection-over-union of at least 0.5 for duration events and ±2 seconds for point events, with boundary-error distributions and stricter sensitivity checks reported alongside. These are proposed protocol choices.
Error severity matters. A slightly late timestamp, an incorrect object, a reversed action and an unsupported claim about a medication event have different consequences. Reports separate boundary errors, missing events, false events, incorrect state changes, identity errors and invented scenarios. High-consequence facts receive human review before being used in a consequential workflow.
EgoSAT evaluates egocentric questions using only previously observed frames and finds confidence can fail to reflect answerability. Our causal replay will ask about completed, ongoing and anticipated events at fixed cutoffs. Reviewers label whether each question is answerable from that prefix. Forecasts remain explicitly inferred. We score answerable-query accuracy, unsupported answers and appropriate abstention separately.
VidHalluc evaluates action, temporal-order and scene-transition hallucinations in video models. We will create analogous paired tests on eligible target footage, including putting in versus taking out, opening versus closing, and a task that starts but is never completed. We also test distractor objects and queries whose answer is absent.
A confidence number must earn its meaning on held-out data. Calibration research and selective prediction motivate measuring error as the acceptance threshold changes. We will tune any confidence model on validation data, publish precision-recall and risk-coverage curves, and retest after domain changes. Abstention improves accepted-output quality only while missed-event accounting remains visible.
10. Make control part of the event system
First-person recordings expose the wearer and other people. The research collection needs informed permission, clear recording boundaries and a way to exclude private intervals before upload. The application should show capture state and support pause, review and deletion. Collection should avoid gathering sensitive material that the intended task does not require.
The proposed architecture separates permission to store a personal memory, permission to share selected context with an assistant and permission to use recordings for training. Each event inherits source access controls. Retrieval is limited to the requesting user and authorized scope. Raw clips, thumbnails, embeddings, summaries and cached answers need coordinated retention and deletion rules.
Training requires its own lineage. Deleting a recording from a personal event store is different from removing its effect from a trained model. We will record which training runs consumed each sample, support exclusion from future runs, and state any limits on removing influence from released weights. Sensitive memories are not silently added to a reusable training set.
An assistant receives only the minimum context required for the user’s task. A proposed reminder or calendar entry remains a draft until the user’s chosen approval policy permits it. Corrections should be easy to make, and a user should be able to inspect which event and footage led to an answer.
11. Use three weeks to earn the next experiment
The first three weeks are a feasibility study for a small ML team, supported by AI-assisted coding, annotation triage and experiment analysis. The objective is an executable benchmark, a recorded-clip baseline and a documented improvement attempt. Reaching the 90% recall objective is a later acceptance gate if the pilot shows a credible route.
One scientist owns target-camera data and annotation quality, one owns models and temporal inference, and a third, if available, owns evaluation, systems profiling and reproducibility. AI tools can draft adapters, generate test cases and summarize logs. Humans verify the split, labels, results and decisions.
Every experiment logs data hashes, preprocessing, sampled timestamps, prompt, model revision, decoding settings, threshold, hardware, wall-clock time and cost. The report includes failed runs and regressions. The final test set is used after candidate selection; further iteration requires a fresh held-out test for a new headline claim.
| Period | Work | Decision artifact |
|---|---|---|
| Days 1–3 | Freeze event definitions, permissions and split groups; collect a small diverse glasses sample | Annotation guide, data manifest and benchmark scope |
| Days 4–7 | Review the 300-clip pilot; run fixed pretrained baselines; inspect continuous-session false alarms | Baseline precision, recall, coverage and failure taxonomy |
| Days 8–12 | Audit one cleared external-data slice; build its time mapping; compare sampling and context ablations | Evidence identifying the dominant missed-event causes |
| Days 13–16 | Run a limited adaptation; compare against the frozen baseline on validation data | Quality-versus-cost result and selected acceptance threshold |
| Days 17–21 | Run final held-out evaluation and causal replay; profile a phone/cloud path; test evidence retrieval | Reproducible report and a go, revise or stop decision |
12. Scale only when quality and economics justify it
The first cost unit is dollars per accepted labeled hour for data and dollars per processed video hour for inference. Both include discarded work. A GPU hourly rate is only one input. Decoding, frame sampling, model size, batching, idle time, retries, storage, network transfer and human review determine the delivered cost. Each run needs a fixed spending limit and checkpoint recovery.
For a concrete three-week compute envelope, Runpod’s listed Pod rates checked on November 4, 2026 are $1.79 per GPU-hour for A100 80 GB, $3.99 for H100 SXM 80 GB and $0.99 for RTX 6000 Ada 48 GB. We propose A100 time for 2B/4B adaptation, H100 time for 8B reference runs and one selected adaptation, and Ada time for baseline inference, label proposals and evaluation.
The budgets below are alternative GPU-only spending caps. Each reserves 20% of the envelope for retries and keeps a small unallocated balance. They exclude personnel, annotation, storage, transfer and API fees. A two-hour profiling run must establish throughput and memory use before the remaining allocation is scheduled. More GPU-hours do not imply a proportional improvement in quality.
Three weeks contain 504 wall-clock hours. The proposed schedules use up to 2, 5 or 10 concurrent independent GPU jobs for the respective tiers, planned within 400 hours to leave scheduling margin. This is a queue of separate experiments. Runtime, availability and checkpoint recovery determine which experiments finish within the cap.
The next data tranche should address an observed failure, such as brief actions, small objects, outdoor glare or a missing task category. The next model run should test a specific intervention. A larger corpus earns its cost when the same benchmark shows better recall at the precision floor, or equivalent quality at lower latency and cost.
Commercial experiments follow a narrow useful workflow. We will test whether people return to their activity history, whether a developer can complete a task with evidence-backed context, and whether an enterprise customer values a specified dataset. Paid adoption, retention, support burden and contribution margin determine whether subscriptions, API usage or custom collection can scale.
Our longer-term precision ambition is 99.99% for accepted events under a published scope, alongside strong recall. That means at most one incorrect accepted event per 10,000 on average under those conditions. Demonstrating it takes far more evidence than the initial pilot. As an illustration, observing zero errors in roughly 30,000 independent accepted events is needed for a one-sided 95% binomial upper error bound near 0.01%. Real events are correlated, so evaluation must account for sessions and domain shifts. The number is a research ambition, not a current result.
The success condition is a system that recovers most useful observable events, shows the evidence, recognizes its limits and helps users complete real tasks at a sustainable cost. A catalog, a fluent caption or a large training run is one component of that work. The deliverable is dependable context another assistant can use.
| GPU envelope | A100 / H100 / Ada hours | Planned GPU spend | Retry reserve | Unallocated |
|---|---|---|---|---|
| $2,000 | 400 / 160 / 200 | $1,552.40 | $400 | $47.60 |
| $5,000 | 800 / 480 / 600 | $3,941.20 | $1,000 | $58.80 |
| $10,000 | 1,600 / 960 / 1,200 | $7,882.40 | $2,000 | $117.60 |
Research and implementation references
- Ego4D — first-person perception and benchmark tasks
- Ego4D episodic memory — evidence localization and moment retrieval
- Ego4D Goal-Step — hierarchical procedural activity annotations
- Ego4D annotation schemas — source clocks and provenance
- EPIC-KITCHENS-100 — Rescaling Egocentric Vision
- EPIC-KITCHENS-100 annotation repository
- Ego-Exo4D atomic descriptions — visibility and uncertainty fields
- EgoLife — egocentric life assistance and memory retrieval
- RekaDaily processed — data, label provenance and consent documentation
- Qwen3-VL technical report — video timestamps and model design
- Qwen3-VL — official implementation and training tools
- VidHalluc — action, temporal-sequence and scene-transition errors
- On Calibration of Modern Neural Networks
- Selective Classification for Deep Neural Networks
- Meta Wearables developer FAQ
- Meta Wearables Device Access Toolkit for iOS
- Ego4D privacy and ethics
- EgoSAT — streaming answerability and confidence
- EgoProbe — long-range and event-specific memory
- Qwen3-VL fine-tuning guide — low-rank adaptation
- Runpod GPU pricing — checked November 4, 2026