Deepfake Temporal Fingerprint Detection

This project implements a baseline deepfake detector that combines spatial features (CNN backbone) with temporal aggregation over sampled video frames. It also includes a FastAPI inference server and a Chrome extension that detects videos on a page and displays deepfake analysis results.

🌐 ML Model API: https://deepfake-temporal-fingerprint-detection.onrender.com

Live Preview & Explainability

---

The Problem

• Deepfake Proliferation: Synthetic videos can be generated at scale and spread rapidly across platforms.
• Verification Cost: Manual forensic inspection is time-consuming and requires specialized expertise.
• Trust & Accountability: Systems must provide confidence and evidence to support decisions.
• Real-World Complexity: Videos vary widely in compression, resolution, lighting, and motion.

This repository addresses these challenges with a practical end-to-end baseline: a trainable model, an API, and a browser extension interface for fast experimentation and demos.

---

Core Features

1. Video Deepfake Detection

• Spatio-Temporal Baseline: ResNet18 feature extraction per frame + temporal average pooling over a short clip.
• Confidence Scoring: Returns probability and confidence to support decision-making.
• FastAPI Inference: Simple endpoints for health/model info and prediction.

2. Explainable AI (XAI)

• Grad-CAM Visuals: Generates heatmaps highlighting regions that influenced the prediction.
• Sanity Checking: Helps verify the model focuses on meaningful facial/video cues rather than artifacts.

AI Model Performance & Architecture

Model Performance Metrics for best model: RESNET18 + TEMPORAL AVG

• Accuracy: 83% - Validation accuracy on the prepared split
• Precision: 84% - Balanced performance across classes
• F1-Score: 83% - Stable trade-off between precision and recall
• ROC-AUC: 0.8905 - Strong discriminative ability

Model Architecture Details

• Base Model: ResNet18 (Transfer Learning from ImageNet)
• Temporal Aggregation: Mean pooling over T sampled frames
• Classifier Head: Linear(512) -> Linear(1) (binary logit)
• Training Strategy: AdamW optimizer
• Loss Function: BCEWithLogitsLoss

Technical Specifications

• Input Resolution: 224x224 pixels
• Frames Sampled: 16 frames per video clip
• Output Classes: 2 (Real, Fake)
• Artifacts: Best checkpoint saved as phase1_best_resnet_temporal_avg.pth

Performance Validation

• Validation Split: Stratified split (default 80/20)
• Metrics: Confusion matrix, classification report, ROC, Precision-Recall
• Explainability: Grad-CAM overlay visualization
• Reproducibility: Fixed random seed for consistent splits

---

Tech Stack

Component	Technologies Used
Browser Extension	Chrome Extension (Manifest v3), JavaScript, DOM Overlay UI
Backend API	Python, FastAPI, Uvicorn
Deep Learning	PyTorch, Torchvision, Transfer Learning (ResNet18)
Explainability	Grad-CAM, Matplotlib
Video Processing	OpenCV, NumPy
Version Control	Git, GitHub

---

Project Structure

Repository Organization

• ml_model/ - Training notebook(s), inference code, and FastAPI server
• extension/ - Chrome extension (popup UI + content script overlays)

Key Components

• ml_model/notebooks/deepfake_detection.ipynb: Training + evaluation + Grad-CAM
• ml_model/api/main.py: FastAPI app with health/model/predict endpoints
• ml_model/src/inference/predict.py: Prediction utilities and model wrapper
• extension/public/content.js: Detects videos and renders overlay badges + side panel

---

Getting Started Locally

Prerequisites

• Python 3.9+
• Node.js (v16+)
• Git

Backend API Setup

1. Install dependencies:

   pip install -r ml_model/requirements.txt

2. Start the FastAPI server:

   uvicorn ml_model.api.main:app --reload

3. API will be available at http://localhost:8000

Chrome Extension Setup

1. Open Chrome extensions page:

   chrome://extensions/

2. Enable Developer mode
3. Load unpacked and select the extension folder

---

API Endpoints

Health & Status

• GET /api/health - System status check and model readiness
• GET /api/model - Model metadata

Inference

• POST /api/predict - Submit a video and get label + probability + confidence

---

Future Enhancements

• Stronger Temporal Modeling: Replace mean pooling with TCN/GRU/Transformer temporal encoders.
• Face Alignment Pipeline: Add face detection + alignment for more stable rPPG and texture cues.
• Robustness: Train across multiple compression levels and mixed deepfake methods.
• Production Extension Flow: Enable real backend inference from the extension with batching and caching.
• Better Explainability: Per-frame CAMs and temporal saliency over the sequence.

---

Contributing

Contributions are welcome. Feel free to open issues for bugs, request features, or submit pull requests to improve the model, API, or extension UX.

---