Troubleshooting¶
Administrative commands reference¶
| Action | Native/PM2 | Docker |
|---|---|---|
| App status | pm2 status |
docker compose -f docker-compose.prod.yml ps |
| Backend logs | pm2 logs itsm-backend |
docker compose -f docker-compose.prod.yml logs -f backend |
| Frontend logs | pm2 logs itsm-frontend |
docker compose -f docker-compose.prod.yml logs -f frontend |
| AI service logs | pm2 logs itsm-ai |
docker compose -f docker-compose.prod.yml logs -f ai |
| Restart everything | pm2 restart all |
docker compose -f docker-compose.prod.yml restart |
| Redis status | sudo systemctl status redis-server |
docker compose -f docker-compose.prod.yml logs redis |
| Nginx status | sudo systemctl status nginx |
docker compose -f docker-compose.prod.yml logs nginx |
| Postgres status | sudo systemctl status postgresql |
docker compose -f docker-compose.prod.yml logs db |
ClamAV (virus scanning)¶
Attachment uploads can be scanned before storage (Settings → Security → Attachment Upload → "Scan Uploads for Malware"). Off by default; when enabled but the daemon is unreachable, uploads fail closed (rejected with 503) rather than being silently let through unscanned.
Install (Ubuntu/Debian, native)¶
sudo apt-get update
sudo apt-get install -y clamav clamav-daemon
# Fresh installs ship a placeholder "Example" line in both config files —
# clamd refuses to start until it's removed:
sudo sed -i '/^Example/d' /etc/clamav/clamd.conf /etc/clamav/freshclam.conf
sudo systemctl enable --now clamav-freshclam # downloads + auto-updates the virus DB
sudo systemctl restart clamav-daemon
sudo systemctl enable clamav-daemon
Use the Unix socket, not TCP¶
Point the app at the Unix socket (/run/clamav/clamd.ctl), not a TCP host/port. Debian/Ubuntu's clamav-daemon package is built around socket-activation with a hard-coded Requires=clamav-daemon.socket — fighting that for a reliable TCP listener isn't worth it (systemctl disable on the socket unit doesn't stop it being pulled back in; only mask does, and that then breaks the service too since it can no longer satisfy Requires=). TCP is only the right call when clamd runs in a separate container/host with no shared filesystem (e.g. docker-compose --profile antivirus).
Verify before enabling in-app:
sudo systemctl status clamav-daemon # should show "active (running)"
sudo ss -xlnp | grep clamd # should show /run/clamav/clamd.ctl
Then: Settings → Security → Attachment Upload → set ClamAV (clamd) Unix Socket Path to /run/clamav/clamd.ctl, enable, save. Nothing on the PM2 side needs restarting — the backend checks the daemon fresh on every upload.
clamd keeps its ~1–2 GB signature database resident in memory — make sure the server has that headroom before enabling.
Nginx version and config layout¶
A fresh native install pulls Nginx from nginx.org's mainline repo (pinned to 1.29.x) rather than the older version your distro's default apt repo ships. An existing install is never silently touched.
If you ever install/upgrade Nginx by hand
Following nginx.org's install instructions directly gives you their stock nginx.conf, which includes /etc/nginx/conf.d/*.conf — not /etc/nginx/sites-enabled/*, the layout Debian/Ubuntu's own package uses and this project's site config depends on. Result: Nginx runs (correct version, nginx -t even passes) but the app's site config isn't loaded, so the app becomes unreachable behind an otherwise-healthy Nginx. install-onprem.sh handles this automatically; a manual install needs include /etc/nginx/sites-enabled/*; added back into the http {} block.
AI microservice¶
itsm-ai is stateless, has no database access or license checks, and is never exposed to the internet. If it's down, the rest of the app keeps working — only AI-powered suggestions (summarization, triage, RCA drafting, SLA risk prediction) stop appearing. Check pm2 logs itsm-ai / docker compose logs ai and confirm LLM_PROVIDER + the matching API key are set in .env.
Docker volumes¶
docker compose down does not delete data — postgres_prod_data, redis_prod_data, and itsm_uploads are named volumes that persist. Only docker compose down -v removes them; never run that against a production stack without a fresh backup in hand.