# AMS technical and research plan ## Track 1 — Reliable local execution and technical foundations **Goal:** establish an independently operable system, measure complete local operation, and provide a trustworthy base for architectural changes. The CTO owns technical decisions and release readiness. ### 1. Ownership and reproducible operation - Verify company-controlled repositories, relevant history, model artifacts, infrastructure, secrets, access and billing. - Identify the authoritative release, runtime configuration, prompts, rule versions and feature flags. - Demonstrate deployment, backup restoration, recovery, rollback and a complete case using the receiving team. - Establish project-scoped access without disturbing unrelated workloads. ### 2. Evaluation and reference data - Inventory and deduplicate at incident level, including re-encodings, alternate camera views and repeated exports. - Separate training, tuning, regression and untouched final evaluation incidents. - Maintain distinct annotations for visible evidence, inferred facts, reference-rule interpretation, operational choice and negotiated/adjudicated outcome. - Use independent review and adjudication; explicitly label ambiguity, insufficient evidence and non-applicable questions. - Evaluate unseen cameras, difficult conditions and case types, not only random splits. - Track annotation agreement, label revisions and performance against the same versioned reference. ### 3. Full local workflow - Evaluate local report extraction, visual reading, structured decisions, escalation, chat, retrieval and explanations. - Compare general models, specialist models and deterministic rule/code implementations for suitable subproblems. - Measure quantization effects, memory, batching, cold starts, peak queues, p50/p95 latency and full operating cost. - Test bounded tool execution, schema validation, retries, timeouts, cancellation and recovery. - Verify operation with outbound inference blocked, including monitoring, retrieval and model-update procedures. ### 4. Decision reliability - Evaluate the Judge on expert-confirmed facts to isolate decision failures from perception failures. - Test chart selection, clause matching, A/B vehicle-role binding, modifiers, abstention and repeat-run consistency separately. - Investigate verified tree-to-verdict regressions, aggregation problems, fallback behavior and sensitivity to one incorrect fact. - Treat unknown and contradictory evidence explicitly rather than forcing a final answer. ### 5. Evidence and state integrity - Preserve source identity, timestamps, evidence references, uncertainty and observed/measured/claimed/inferred/human-confirmed status across every component. - Support conflicting corrections, versioned case state, explicit approval ownership and concurrent access. - Ensure reanalysis cannot silently override approved decisions. - Invalidate stale explanations when the underlying approved facts or ratios change. - Make every material output reproducible from its recorded inputs and versions. ### 6. Versioned domain semantics - Define temporal meanings such as preceding/following, stopping/starting and lane-change initiation/crossing/completion. - Record reference definitions, partner-specific interpretations, exceptions and approval status separately. - Associate each definition with positive, negative and ambiguous examples. - Test rule consistency, conflicting definitions and the consequences of version changes. ### 7. Confidence and observability - Distinguish confidence in a measurement, a fact, an interpretation and the final decision. - Calibrate estimates against reviewed evidence where feasible; avoid presenting path labels as correctness probabilities. - Measure error at specified accepted coverage, abstention quality and confident wrong answers. - Record model/tool calls, failures, latency and resource cost sufficiently to compare releases. ### 8. Release gates - Reproducible baseline and independently operable deployment. - Agreed technical quality, coverage and performance thresholds. - Verified local boundary, recovery and peak capacity. - Explicit residual limitations and escalation behavior. - Technical approval by the CTO, separate from customer acceptance. ## Track 2 — Technical and foundational research portfolio **Goal:** test improvements at every layer, including new representations and architectures. Existing components may be retained, replaced or rebuilt according to evidence. This lists the complete scope discussed for this plan. It is a research backlog, not a requirement to run every experiment simultaneously or a claim that the field has no other directions. ### 1. Temporal visual representations **Question:** can a learned video representation understand maneuvers more reliably and with fewer labels than frame-by-frame text extraction? - Compare image features with temporal heads, frozen video encoders and domain adaptation. - Evaluate stop-then-go, overtaking, braking and lane-change event sequences. - Compare quality at fixed label budgets and hold out camera families. - Explore self-supervised pretraining on approved unlabeled driving video when simpler frozen-feature methods are insufficient. **Measure:** event accuracy, temporal grounding, unseen-camera generalization, label efficiency, memory and latency. ### 2. Multiple plausible event histories **Question:** can the system avoid an early wrong commitment when vehicle identity, timing or maneuver is ambiguous? - Maintain a bounded set of alternative actor tracks, event intervals and maneuver histories. - Apply temporal, geometric and evidence constraints to eliminate inconsistent hypotheses. - Compare against single-best interpretation and repeated model sampling. - Identify the evidence that would distinguish the remaining histories. **Measure:** correct-history retention, confident errors, accepted coverage, review effort and hypothesis-management cost. ### 3. Active evidence acquisition **Question:** which next observation is worth its collection or processing cost? - Select among another frame/crop, higher-rate sampling, a secondary view, a location field or a human answer. - Compare fixed questionnaires, uncertainty-based selection and value-of-information policies. - Include the cost of human effort, access delay and GPU processing. - Permit stopping when further evidence is unavailable or unlikely to resolve the issue. **Measure:** questions/actions per resolved case, useful evidence gained, time, cost and quality at comparable coverage. ### 4. Joint 3D/4D scene reconstruction **Question:** can joint camera-and-scene estimation improve relative motion and geometry under changing camera conditions? - Compare classical geometry, existing BEV approaches and persistent/dynamic reconstruction. - Separate camera motion from actor motion and preserve identity through occlusion. - Test reconstruction with known scene anchors before evaluating unconstrained clips. - Keep relative geometry separate from metric scale when absolute distance is unidentifiable. **Measure:** camera/trajectory error, scale drift, relative ordering, physical consistency, uncertainty and compute cost. Visual appearance alone is not a success metric. ### 5. Counterfactual avoidability **Question:** would an alternative action have avoided contact across plausible assumptions? - Begin with controlled kinematic simulation and known factual/counterfactual trajectories. - Vary reaction time, braking, friction, visibility and maneuver onset explicitly. - Compare interpretable simulation with learned dynamics only after factual reconstruction is reliable. - Preserve observed constraints; distinguish simulated alternatives from recorded evidence. **Measure:** outcome error, sensitivity, factual consistency and expert review. Do not equate physical avoidability with legal fault automatically. ### 6. Counterpart identification and tracking - Compare candidate generation, candidate ranking and joint identification across time. - Test identity switches, occlusion, partial visibility, reflections and multiple nearby vehicles. - Evaluate ego/other-camera ownership and explicitly allow the counterpart to be unobservable. - Compare appearance, motion and interaction cues rather than relying on box size alone. **Measure:** correct counterpart, track continuity, identity switches and downstream decision errors. ### 7. Ego motion and camera calibration - Compare fixed priors, estimated intrinsics/extrinsics and self-calibration across clips. - Detect camera movement caused by mounting changes, bumps or collision. - Separate pre-impact maneuver from impact-induced motion. - Fuse approved speed/GPS measurements where available and quantify synchronization error. **Measure:** motion direction, yaw, displacement error where measurable, calibration stability and uncertainty. ### 8. Impact localization and event boundaries - Fuse visual motion, contact cues, audio and available metadata. - Test long clips, multiple impacts, near misses and impact outside the field of view. - Estimate event intervals rather than only a single timestamp. - Use distinct windows for maneuver onset, approach, impact and post-impact motion. **Measure:** onset/impact timing error, missed events and downstream interpretation sensitivity. ### 9. Lane geometry, markings and topology - Distinguish lane identity, lane boundaries, solid/dashed markings, color, junction topology and road-edge/safety-zone structure. - Track lane geometry through occlusion rather than treating every frame independently. - Estimate lane crossing and initiation separately. - Test map-assisted interpretation with map freshness and incident-time disagreement checks. **Measure:** marking classification, lane identity, crossing/onset error and consequences for rule-relevant facts. ### 10. Turn signals, brake lights and traffic controls - Identify lamps and distinguish indicator blinking, hazards, braking, reflection and compression artifacts. - Estimate signal start/end relative to maneuver onset and visible reaction opportunity. - Associate a traffic light or sign with the relevant lane and actor. - Compare task-specific detectors, segmentation/tracking and video-model interpretations. **Measure:** actor association, phase/onset accuracy, visibility-aware abstention and modifier-evidence quality. ### 11. Temporal semantics and rule reasoning - Represent predicates over intervals rather than isolated labels. - Compare explicit rules, learned structured decisions and hybrid constraints. - Test contradiction detection, sibling-case distinctions, exceptions and unknown conditions. - Explore formal checks and executable examples for rule changes. **Measure:** decisions on expert-confirmed facts, consistency, role correctness and sensitivity to ambiguous language. ### 12. Uncertainty, calibration and out-of-distribution detection - Calibrate perception and decision estimates separately. - Detect unfamiliar cameras, locations, lighting and unsupported event types. - Compare uncertainty-based referral, ensembles and prediction-set approaches. - Distinguish missing evidence from conflicting evidence and model uncertainty. **Measure:** calibration, confident errors, error/coverage curves, referral burden and performance under distribution shift. ### 13. Multi-camera and multimodal fusion - Synchronize front, rear, counterpart and CCTV views with clock offset/drift and different frame rates. - Match actors across views and preserve view-specific visibility. - Fuse video, audio, GPS, speed, reports and approved external road information. - Preserve conflicts and source dependencies rather than treating correlated reports as independent confirmation. **Measure:** alignment error, cross-view identity, added evidence recall, conflict resolution and marginal value per added source. ### 14. Video and damage-photo consistency - Compare claimed contact locations, vehicle orientation and visible damage with video-derived contact hypotheses. - Distinguish pre-existing damage, occluded damage and uncertain correspondence. - Start with independently linked incident photos and expert contact annotations. - Treat disagreement as a review signal, not proof of fraud or responsibility. **Measure:** contact-region agreement, false inconsistency alerts, review usefulness and robustness to missing photos. ### 15. Training, distillation and specialization - Compare supervised tuning, lightweight adapters, distillation and specialist task heads. - Train evidence extraction and structured decisions separately where the evaluation supports that division. - Compare explicit orchestration with model-selected tool calls; bound repeated calls and malformed outputs. - Test multilingual terminology, quantization and deployment-aware compression. - Compare domain-adapted pretrained models with a small custom temporal model when data and compute justify it. Large-scale foundation pretraining is a separately scoped option, not an assumed requirement. **Measure:** unseen-incident quality, teacher/student gap, label efficiency, tool-call reliability, memory and end-to-end cost. Use approved training data and teacher outputs. ### 16. Synthetic data and controlled simulation - Generate controlled variations in camera placement, event ordering, occlusion, signals, lighting and road geometry. - Produce ground truth for timing, trajectories and alternative actions where real labels are difficult. - Test whether synthetic augmentation improves real held-out incidents. - Separate synthetic data from final real-world evaluation and detect simulator-specific shortcuts. **Measure:** real-data gain, rare-condition coverage, annotation savings and simulation-to-real transfer. ### 17. Robustness to degraded and incomplete footage - Test low resolution, low light, rain, glare, blur, compression, variable frame rate, dropped frames and short/truncated clips. - Compare quality-aware sampling, preprocessing and task-specific enhancement. - Verify whether enhancement preserves evidence or invents details; retain original frames. - Test graceful failure and evidence-quality reporting. **Measure:** fact degradation, false detail creation, uncertainty response and usable coverage. ### 18. Visual case retrieval and outcome-linked reasoning - Retrieve incidents by actor interactions, geometry and temporal events, not just textual similarity. - Separate visual similarity, rule relevance and agreement/outcome similarity. - Test retrieval of genuinely comparable cases with explicit supporting and differing facts. - Treat outcomes without reasons as incomplete supervision and evaluate partner-specific effects separately. **Measure:** expert-rated relevance, misleading favorable examples, evidence traceability and decision/review benefit. ## Experiment management and sequencing - Every experiment needs a hypothesis, baseline/comparator, incident-separated data, evaluation metrics, owner, fixed initial budget and stop/continue criterion. - Start with the baseline/diagnostic work in Track 1. Prioritize research according to measured error contribution and data availability. - Initial foundational candidates: temporal representations and multiple event histories. Active evidence acquisition is an alternative when labeled temporal data is limited. - Run targeted CV experiments where diagnostics identify a specific bottleneck; do not wait for a new general model to fix a measurable component failure. - Start reconstruction feasibility only with suitable geometry evaluation. Make multimodal fusion and damage consistency conditional on linked data access. - Keep counterfactual analysis and large-scale pretraining outside the initial release dependency. - Support a limited number of concurrent experiments based on real staffing. Inclusion in the portfolio does not imply immediate funding. - Continue when held-out evidence demonstrates useful quality, coverage, effort or cost improvements. Stop or revise when gains disappear under new cameras, depend on leaked data, or cannot justify complexity. ## Questions for CEO 1. What is the priority order between local deployment, improving quality and foundational research? 2. What scope, dates and acceptance criteria have already been committed to KB? 3. What are the budget and staffing envelopes for delivery and protected research? Can we assign named owners now? 4. Can you secure KB's written deployment constraints, technical contacts and an evaluation environment inside its network? 5. Can you obtain a representative linked data sample, a data owner and a delivery date before we estimate data-dependent work? 6. Who at KB can approve disputed definitions and reference labels, and how much expert review time will they commit? 7. What quality, coverage, latency and peak workload must we support? Which trade-offs can we negotiate? 8. Which additional inputs are available: other camera views, photos, audio, location, speed, structured reports and outcome reasoning? 9. What data and model reuse rights should we assume for training and later releases? 10. What research investment is available before customer funding, including compute, annotation and expert review? 11. If access, data or local quality gates slip, should we reduce scope, move dates or increase resources—and who approves that change? ## Immediate actions - **CTO:** assign owners, prioritize the research portfolio and approve experiment gates. - **Engineering:** prove independent operation, freeze the baseline and identify measured failure categories. - **ML/evaluation:** prepare data partitions, reference annotations and component-level evaluation. - **CEO:** resolve scope, resources and KB access/domain-review dependencies. - **Joint review after two weeks:** assess evidence, blockers and spending; decide what to retain, replace, defer or stop.