Final Research Project | Multimodal AI

celestx.

Spotting the clouds, before the storm

A multimodal AI system for early depression detection in Indonesian social media. IndoBERTweet reads the text, a Vision Transformer reads the images, and the two fuse into one clinical risk profile.

View on GitHub ↑Watch demo ↓
Solo author2 supervisors9 DSM-5 symptoms34 commits

82.10%

accuracy · late fusion, 5-fold CV (294 pairs)

79.10%

F1-score · multimodal ViT fusion

13

false negatives · of 294 cases

SUS 80

usability score · 14 respondents (Excellent)

Demo · recorded run

See the dashboard run.

The real stack is FastAPI plus PyTorch on a local machine. Below is a recorded run, and the exported predictions to poke at. The model is not served publicly.

Recorded locally · silent screen capture

Exported posts

@sample-01

20 Apr 2026

INDICATED

Pada depresi, pola tidur bisa berubah jadi 2 arah hipersomnia (tidur berlebihan) & insomnia (susah tidur). Laki-laki lebih sering “withdraw” → tidur terus,. perempuan lebih “overthinking” → susah tidur. Tapi ini ga mutlak ya

Model confidence99%

DSM-5 category

Negative Mood

Suasana Hati Negatif

Matched keyword

"depresi"

First hit in the app's lexicon order

Tawarkan pendekatan empati personal, ajak komunikasi tanpa menghakimi.

Handles and avatars are masked, post text is unchanged. The archive only exports posts the model flagged, so every sample reads INDICATED.

Landing Page

Dashboard

History Page

XAI Section

XAI Section

Deep Scan

Profile Page

Profile Page

Profile Page

Tech Stack

1IndoBERTweet
2Vision Transformer
3PyTorch
4FastAPI
5React.js
6Tailwind CSS
7Framer Motion
8SHAP
9Chrome Extension
10Docker
11Git

The brief · problem → approach → result

01 / The problem

Depression signals in Indonesian social media go unnoticed until they escalate. Text-only models miss what images carry, and clinical frameworks rarely make it into the model itself.

02 / The approach

Fuse fine-tuned IndoBERTweet (text) and a Vision Transformer (image) into one risk profile, mapped to 9 DSM-5 symptoms across 3 severity thresholds, explained through SHAP values and ViT attention maps.

03 / The result

A working product end to end: React dashboard, Chrome extension scoring posts in real time, FastAPI inference, Docker deploy. The text model was trained on a balanced 7,991-tweet corpus; the multimodal fusion was evaluated with 5-fold CV on 294 tweet-image pairs, reaching 82.10% accuracy and 79.10% F1 with 13 false negatives out of 294 cases. The first-author paper was presented at ICAICTA 2026.

Two Audiences. One System.

Dual-persona design|Index / 01

Persona A

Students & General Users

Use CelestX as a digital exposure meter. Track warning signals on your own timeline or that of someone close to you. The Chrome Extension reads text and images on Twitter/X in real time and shows alerts without leaving the platform.

Self-monitoringChrome ExtensionReal-time alerts

Persona B

Psychologists & Clinicians

CelestX works as a digital phenotyping tool. Collect behavioral signals from a client's social media as a clinical dashboard, a continuous risk archive, and model explanations powered by SHAP and ViT attention maps.

Clinical dashboardRisk archiveSHAP explainability

Core Features

Command | Archive | Insights|Index / 02
⌘

Command Center

Central dashboard for monitoring every profile under observation. Real-time risk timeline, threshold notifications, and a DSM-5 category summary on a single screen.

  • • Real-time monitoring panel
  • • Multi-profile management
  • • Threshold alert engine
🗄

Archive Vault

A structured repository for every screening result. Comes with SHAP values that pinpoint the words and phrases that drove the decision, and ViT attention maps that highlight the image regions that influenced it.

SHAP explainabilityViT attention mapsSearch & filter
◎

Insights Hub

Clinical analytics for advanced users: weekly and monthly risk trends, category distribution, coarse demographics (anonymized), and PDF report export for clinical discussions.

Trend analyticsExport reports (PDF)Anonymized cohorts

DSM-5 Clinical Mapping

9 symptoms | 3 thresholds|Index / 03

9 DSM-5 Symptoms Mapped

  • 01Depressed mood
  • 02Anhedonia (loss of interest)
  • 03Appetite changes
  • 04Sleep disturbance
  • 05Psychomotor agitation
  • 06Fatigue
  • 07Feelings of worthlessness
  • 08Concentration loss
  • 09Suicidal ideation

3-Tier Risk Threshold

S

STABLE

0-14%

Normal emotional fluctuation. No clinical concern.

M

MODERATE

15-30%

Early onset of psychological distress. Monitor closely.

H

HIGH

> 30%

Critical depressive prevalence. Requires professional attention.

Design principle: the system is tuned for high recall. A false positive that gets reviewed by a human beats a false negative that misses a signal entirely.

Research prototype and decision-support concept, not a clinical diagnostic tool.

Technology Architecture

Modal fusion pipeline|Index / 04
📝

Text

IndoBERTweet fine-tuned

📄

Image

ViT feature extraction

⚙

Fusion

late fusion · 1536-d concat

📤

Output

risk score + SHAP + attention map

celestx/
├── backend/                    # python fastapi server
│   ├── main.py                 # api entry point & routing
│   ├── unzip_models.py         # script to combine & extract model files
│   ├── requirements.txt        # backend dependencies
│   └── ...                     # utility scripts
├── dashboard/                  # react (vite) frontend & extension
│   ├── src/                    # react source code
│   │   ├── pages/              # dashboard, history, analysis pages
│   │   ├── components/         # reusable ui components
│   │   └── constants/          # clinical lexicon (dsm-5)
│   ├── manifest.json           # chrome extension manifest v3
│   ├── background.js           # extension background logic
│   └── ...                     # build configurations
└── push.bat                    # deployment utility script