Neuromorphic Camera Modeling

Problem

Event-based (neuromorphic) cameras are a fundamentally different kind of image sensor. Unlike conventional frame-based cameras, each pixel independently fires an asynchronous “event” when its local brightness change exceeds a threshold. This produces a high-temporal-resolution stream of events rather than a sequence of frames. The challenge is that these sensors generate events from both genuine brightness changes (signal) and internal noise processes, and existing models treat these two sources separately or incompletely. Without a unified statistical description, it is difficult to calibrate the sensor, predict its behavior under novel conditions, or design downstream algorithms that properly account for noise.

Physical System

The sensor under study is an event-based camera with independently operating pixels. Each pixel monitors local log-intensity and fires events when changes cross positive or negative contrast thresholds. The relevant physics includes photon arrival statistics, electronic readout noise, threshold variability across pixels, and refractory timing. The model must describe both the signal pathway (brightness changes that produce events) and the noise pathway (spontaneous events from thermal and electronic noise) within a single probabilistic framework.

What I Built

I developed an analytical statistical model that unifies signal-driven and noise-induced event generation into one probabilistic description. The model characterizes the event rate as a function of physical sensor parameters (contrast thresholds, noise levels, refractory period, and background activity rate) and input stimulus properties. This formulation allows the model to:

• Predict event rates and timing distributions for arbitrary input stimuli
• Separate signal contributions from noise contributions in recorded event streams
• Serve as a forward model for calibrated sensor simulation
• Provide a likelihood function for parameter inference from real data

Experimental & Computational Workflow

1. Model derivation: Derived closed-form expressions for event-generation probabilities as functions of physical sensor parameters and stimulus conditions.
2. Controlled recordings: Designed experimental protocols to record event streams from the camera under known stimulus conditions, producing datasets with ground-truth input signals.
3. Parameter estimation: Built an inference workflow that fits the model to recorded event streams, extracting estimates of internal sensor parameters (contrast thresholds, noise rates, refractory timing) from the data.
4. Stability validation: Tested whether inferred parameters remain stable across different experimental conditions, verifying that the model captures genuine sensor properties rather than overfitting to individual recordings.
5. Cross-validation with deep learning: Used the companion deep-learning reconstruction project (U-Net) as an independent validation signal: if the model correctly describes the sensor, images reconstructed from model-predicted noise statistics should match reconstructions from real noise.

Validation

The model is validated through multiple independent checks:

• Parameter stability: Inferred sensor parameters are consistent across different experimental conditions and recording sessions.
• Predictive accuracy: The model predicts event-rate distributions that match empirical distributions from held-out recordings.
• Reconstruction consistency: Deep-learning reconstructions trained on model-guided data produce outputs consistent with reconstructions from raw sensor data.

Key Outcomes

Relevance

This project demonstrates core competencies in sensor modeling, statistical inference from sensor data, experimental design for calibration, and computational optics. The workflow—from physical model derivation through parameter estimation to experimental validation—directly parallels the process of evaluating and characterizing electro-optical and infrared sensor systems in defense and aerospace contexts.