feat: Implement image fetching and storage system
Some checks failed
Deploy to BeePC / deploy (push) Has been cancelled
Some checks failed
Deploy to BeePC / deploy (push) Has been cancelled
- Add image-fetcher module for downloading and saving images from various sources. - Create storage module for managing image files, including downloading, verifying integrity, and cleaning up orphaned files. - Develop gallery HTML page for displaying images with sorting and filtering options. - Set up RESTful API routes for image management, including fetching, adding tags, and deleting images. - Introduce setup script for initializing the database and configuring image sources. - Implement a batch process for verifying image integrity and cleaning up old images. - Add setup batch script for easy installation and configuration of the image storage system.
This commit is contained in:
412
DOCKER_DEPLOYMENT.md
Normal file
412
DOCKER_DEPLOYMENT.md
Normal file
@@ -0,0 +1,412 @@
|
||||
# 🐳 Docker Deployment Guide for Image Storage
|
||||
|
||||
This guide covers deployment of the HomeBase image storage system in Docker on your home lab.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Docker and Docker Compose installed on your home lab server (beepc)
|
||||
- SSH access configured (required for deploy script)
|
||||
- PowerShell 5+ or PowerShell Core for running deploy scripts
|
||||
- Network access from your dev machine to the home lab
|
||||
|
||||
## Quick Deployment
|
||||
|
||||
### 1. Deploy to Home Lab
|
||||
|
||||
From your local machine, run:
|
||||
|
||||
```powershell
|
||||
.\scripts\deploy.ps1
|
||||
```
|
||||
|
||||
This script will:
|
||||
- Copy all project files to your home lab server (spencer@beepc)
|
||||
- Build the Docker image
|
||||
- Create and start the container with persistent storage
|
||||
- Set up systemd service for auto-start
|
||||
|
||||
### 2. After Deployment
|
||||
|
||||
The container will automatically:
|
||||
- Create the SQLite database
|
||||
- Create image storage directories
|
||||
- Start the image fetcher (if configured)
|
||||
- Be available at `http://homebase.sketchferret.com`
|
||||
|
||||
## Docker Setup Details
|
||||
|
||||
### Persistent Storage
|
||||
|
||||
Images are stored in a Docker volume called `homebase-data`:
|
||||
|
||||
```yaml
|
||||
volumes:
|
||||
- homebase-data:/app/data
|
||||
```
|
||||
|
||||
This ensures:
|
||||
- Database persists across container restarts
|
||||
- Image files are not lost on container recreation
|
||||
- Data survives container updates
|
||||
|
||||
### Viewing Container Logs
|
||||
|
||||
```bash
|
||||
# SSH into home lab
|
||||
ssh spencer@beepc
|
||||
|
||||
# View container logs
|
||||
docker compose -f ~/homebase/docker-compose.yml logs -f
|
||||
|
||||
# Or use systemd
|
||||
journalctl -u homebase -f
|
||||
```
|
||||
|
||||
### Container Management
|
||||
|
||||
```bash
|
||||
# Stop container
|
||||
docker compose -f ~/homebase/docker-compose.yml down
|
||||
|
||||
# Rebuild and restart
|
||||
docker compose -f ~/homebase/docker-compose.yml up -d --build
|
||||
|
||||
# Check status
|
||||
docker compose -f ~/homebase/docker-compose.yml ps
|
||||
|
||||
# View stored images
|
||||
docker exec homebase ls -lh /app/data/images/
|
||||
```
|
||||
|
||||
## Configuration in Docker
|
||||
|
||||
### Image Sources (image-sources.json)
|
||||
|
||||
After deployment, edit the configuration on the remote server:
|
||||
|
||||
```bash
|
||||
ssh spencer@beepc
|
||||
nano ~/homebase/image-sources.json
|
||||
```
|
||||
|
||||
Then restart the container:
|
||||
|
||||
```bash
|
||||
cd ~/homebase
|
||||
docker compose restart app
|
||||
```
|
||||
|
||||
### Environment Variables
|
||||
|
||||
Edit `docker-compose.yml` to customize:
|
||||
|
||||
```yaml
|
||||
environment:
|
||||
- NODE_ENV=production
|
||||
- PORT=3001
|
||||
- DOMAIN=homebase.sketchferret.com
|
||||
```
|
||||
|
||||
## Managing Data
|
||||
|
||||
### Backup Database
|
||||
|
||||
The SQLite database is stored in the Docker volume at `/app/data/homebase.db`
|
||||
|
||||
```bash
|
||||
# SSH into server
|
||||
ssh spencer@beepc
|
||||
|
||||
# Backup database
|
||||
docker run --rm -v homebase-data:/data \
|
||||
-v ~/backups:/backup \
|
||||
alpine cp /data/homebase.db /backup/homebase.db.$(date +%Y%m%d)
|
||||
|
||||
# Backup images
|
||||
docker run --rm -v homebase-data:/data \
|
||||
-v ~/backups:/backup \
|
||||
alpine tar czf /backup/images.$(date +%Y%m%d).tar.gz /data/images/
|
||||
```
|
||||
|
||||
### Restore Database
|
||||
|
||||
```bash
|
||||
docker run --rm -v homebase-data:/data \
|
||||
-v ~/backups:/backup \
|
||||
alpine cp /backup/homebase.db.20260211 /data/homebase.db
|
||||
|
||||
# Restart container
|
||||
docker restart homebase
|
||||
```
|
||||
|
||||
### Clean Up Old Images
|
||||
|
||||
```bash
|
||||
# SSH into server
|
||||
ssh spencer@beepc
|
||||
|
||||
# Connect to running container
|
||||
docker exec -it homebase node -e "
|
||||
const db = require('./lib/database');
|
||||
db.initializeDatabase().then(async () => {
|
||||
const deleted = await db.cleanupOldImages(30);
|
||||
console.log('Deleted', deleted, 'old images');
|
||||
process.exit(0);
|
||||
}).catch(err => {
|
||||
console.error(err);
|
||||
process.exit(1);
|
||||
});
|
||||
"
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Container won't start
|
||||
|
||||
```bash
|
||||
ssh spencer@beepc
|
||||
cd ~/homebase
|
||||
|
||||
# Check logs
|
||||
docker compose logs app
|
||||
|
||||
# Rebuild
|
||||
docker compose up -d --build
|
||||
|
||||
# Check again
|
||||
docker compose ps
|
||||
```
|
||||
|
||||
### Permission errors
|
||||
|
||||
The container runs as the `node` user. Check logs:
|
||||
|
||||
```bash
|
||||
docker compose exec app ls -la /app/data/
|
||||
```
|
||||
|
||||
All files should be owned by `node`.
|
||||
|
||||
### Can't access via domain
|
||||
|
||||
Check that the service is running:
|
||||
|
||||
```bash
|
||||
ssh spencer@beepc
|
||||
|
||||
# Check systemd
|
||||
systemctl status homebase
|
||||
|
||||
# Check Docker
|
||||
docker ps | grep homebase
|
||||
|
||||
# Test port
|
||||
curl -i http://localhost:3001/health
|
||||
```
|
||||
|
||||
### Database is locked
|
||||
|
||||
This usually means two processes are accessing the database simultaneously:
|
||||
|
||||
```bash
|
||||
# Check if multiple containers are running
|
||||
docker ps -a | grep homebase
|
||||
|
||||
# Stop all containers
|
||||
docker compose down
|
||||
|
||||
# Verify only one is running
|
||||
docker compose up -d
|
||||
```
|
||||
|
||||
## Updating the Deployment
|
||||
|
||||
### Update Code
|
||||
|
||||
After making changes locally:
|
||||
|
||||
```powershell
|
||||
cd c:\Users\srene\Sources\HomeBase
|
||||
git commit -am "Add new features"
|
||||
git push
|
||||
|
||||
# Deploy to home lab
|
||||
.\scripts\deploy.ps1
|
||||
```
|
||||
|
||||
### Update Dependencies
|
||||
|
||||
If you update `package.json`:
|
||||
|
||||
```powershell
|
||||
# From your local machine
|
||||
.\scripts\deploy.ps1
|
||||
|
||||
# The deploy script will:
|
||||
# 1. Copy new package.json
|
||||
# 2. Rebuild the Docker image (npm install runs here)
|
||||
# 3. Restart the container
|
||||
```
|
||||
|
||||
### Manual Updates
|
||||
|
||||
SSH into the server and run:
|
||||
|
||||
```bash
|
||||
cd ~/homebase
|
||||
|
||||
# Pull latest code
|
||||
git pull # If using git
|
||||
|
||||
# Or manually copy files via SCP
|
||||
# Then rebuild:
|
||||
docker compose up -d --build
|
||||
```
|
||||
|
||||
## Systemd Service
|
||||
|
||||
The deployment script sets up a systemd service:
|
||||
|
||||
```bash
|
||||
# Check service
|
||||
systemctl status homebase
|
||||
|
||||
# Start service
|
||||
sudo systemctl start homebase
|
||||
|
||||
# Stop service
|
||||
sudo systemctl stop homebase
|
||||
|
||||
# View logs
|
||||
journalctl -u homebase -f
|
||||
```
|
||||
|
||||
Service file location: `/etc/systemd/system/homebase.service`
|
||||
|
||||
## Volume Management
|
||||
|
||||
### List volumes
|
||||
|
||||
```bash
|
||||
docker volume ls | grep homebase
|
||||
```
|
||||
|
||||
### Inspect volume
|
||||
|
||||
```bash
|
||||
docker volume inspect homebase-data
|
||||
```
|
||||
|
||||
Shows the mount point on the host server.
|
||||
|
||||
### Manual mount (for recovery)
|
||||
|
||||
```bash
|
||||
docker run -it --rm -v homebase-data:/data alpine sh
|
||||
```
|
||||
|
||||
Then you can inspect files at `/data/`
|
||||
|
||||
## Performance Optimization
|
||||
|
||||
### Monitor Storage Usage
|
||||
|
||||
```bash
|
||||
curl http://homebase.sketchferret.com/api/stats
|
||||
```
|
||||
|
||||
### Cleanup Recommendations
|
||||
|
||||
- **Weekly**: Run cleanup for images older than 7 days
|
||||
- **Monthly**: Run verification to check corruption
|
||||
- **Quarterly**: Full database optimization
|
||||
|
||||
```bash
|
||||
# SSH into server
|
||||
ssh spencer@beepc
|
||||
docker exec homebase node -e "
|
||||
const db = require('./lib/database');
|
||||
const storage = require('./lib/storage');
|
||||
|
||||
db.initializeDatabase().then(async () => {
|
||||
console.log('Running cleanup...');
|
||||
const deleted = await db.cleanupOldImages(7);
|
||||
const orphaned = await storage.cleanupOrphanedFiles(db.db);
|
||||
console.log('Deleted:', deleted, 'old images');
|
||||
console.log('Removed:', orphaned.cleaned, 'orphaned files');
|
||||
process.exit(0);
|
||||
}).catch(err => {
|
||||
console.error('Error:', err);
|
||||
process.exit(1);
|
||||
});
|
||||
"
|
||||
```
|
||||
|
||||
## Network & Firewall
|
||||
|
||||
### Open Ports
|
||||
|
||||
Ensure port 3001 is accessible on your home lab network:
|
||||
|
||||
```bash
|
||||
# Check if port is listening
|
||||
ssh spencer@beepc
|
||||
netstat -tlnp | grep 3001
|
||||
|
||||
# Or use Docker
|
||||
docker port homebase
|
||||
```
|
||||
|
||||
### Domain Configuration
|
||||
|
||||
The app is configured to run on `homebase.sketchferret.com`. Ensure:
|
||||
- DNS resolves to your home lab IP
|
||||
- Firewall allows traffic on port 3001
|
||||
- Your home lab router has port forwarding (if external access needed)
|
||||
|
||||
## Debugging
|
||||
|
||||
### Enable detailed logging
|
||||
|
||||
```bash
|
||||
# SSH into server
|
||||
ssh spencer@beepc
|
||||
|
||||
# View container logs with timestamps
|
||||
docker compose logs -f --timestamps app
|
||||
|
||||
# Or view the last 100 lines
|
||||
docker compose logs --tail=100 app
|
||||
```
|
||||
|
||||
### Access container shell
|
||||
|
||||
```bash
|
||||
docker exec -it homebase sh
|
||||
cd /app
|
||||
node setup.js config
|
||||
```
|
||||
|
||||
### Database inspection
|
||||
|
||||
```bash
|
||||
# Access SQLite directly
|
||||
docker exec -it homebase sqlite3 /app/data/homebase.db
|
||||
|
||||
# In sqlite prompt:
|
||||
sqlite> SELECT COUNT(*) FROM images;
|
||||
sqlite> SELECT * FROM tags;
|
||||
sqlite> .quit
|
||||
```
|
||||
|
||||
## Best Practices
|
||||
|
||||
1. **Regular Backups**: Back up the database volume weekly
|
||||
2. **Log Monitoring**: Check logs regularly for errors
|
||||
3. **Storage Limits**: Monitor storage and clean up old images
|
||||
4. **Configuration**: Keep `image-sources.json` updated with working URLs
|
||||
5. **Testing**: Test image fetching after configuration changes
|
||||
|
||||
---
|
||||
|
||||
For API usage, see [IMAGE_STORAGE_GUIDE.md](IMAGE_STORAGE_GUIDE.md)
|
||||
Reference in New Issue
Block a user