feat: Implement image fetching and storage system

- Add image-fetcher module for downloading and saving images from various sources. - Create storage module for managing image files, including downloading, verifying integrity, and cleaning up orphaned files. - Develop gallery HTML page for displaying images with sorting and filtering options. - Set up RESTful API routes for image management, including fetching, adding tags, and deleting images. - Introduce setup script for initializing the database and configuring image sources. - Implement a batch process for verifying image integrity and cleaning up old images. - Add setup batch script for easy installation and configuration of the image storage system.
2026-02-12 13:13:36 -05:00
parent ea6cc3fc85
commit 9c72b00b1b
19 changed files with 3004 additions and 71 deletions
--- a/README.md
+++ b/README.md
@@ -1,13 +1,18 @@
 # HomeBase

-Simple self-hosted Node.js Docker application with automated SSH deployment.
+Self-hosted Node.js Docker application with automated SSH deployment and **long-term image storage** with corruption detection and tagging for ML training.

 ## Features

- 🚀 Simple Express.js server
- 🐳 Docker containerized
+- 🚀 Express.js server
+- 🐳 Docker containerized with persistent volumes
 - 🔄 Automated deployment via SSH
- ✅ Health check endpoint
+- 📸 **Long-term image storage** with SQLite database
+- 🏷️ **Tag-based organization** for machine learning datasets
+- 🛡️ **Corruption detection** using SHA256 checksums
+- 🔍 **RESTful API** for image management
+- 🧪 **Automatic fetching** every 2-3 minutes from web services
+- ✅ Health check endpoints
 - 🔐 SSH key-based authentication

 ## Prerequisites
@@ -30,53 +35,70 @@ On the remote server (beepc):
 # Install dependencies
 npm install

+# Initialize the image storage system
+node setup.js init
+
+# Configure image sources (edit image-sources.json)
+# Set "enabled": true for sources you want to fetch from
+
 # Run locally
 npm start

 # Visit http://localhost:3001
 ```

+### Image Storage Features
+
+Once running, manage images via the REST API:
+
+```bash
+# List all images
+curl http://localhost:3001/api/images
+
+# Add an image manually
+curl -X POST http://localhost:3001/api/images \
+  -H "Content-Type: application/json" \
+  -d '{"source_url":"https://example.com/image.jpg","tags":["test"]}'
+
+# Get storage statistics
+curl http://localhost:3001/api/stats
+
+# Verify all images for corruption
+curl -X POST http://localhost:3001/api/verify
+```
+
+📖 **Full documentation:**
+- [📸 Image Storage Guide](IMAGE_STORAGE_GUIDE.md) - Complete API reference, configuration, and usage
+- [🐳 Docker Deployment Guide](DOCKER_DEPLOYMENT.md) - Deployment to home lab with persistent storage
+
 ### Deployment

-### Option 1: Manual Deployment (Windows)
+### Quick Deployment to Home Lab (PowerShell)

-```bash
-# Run the deployment script from project root
-.\scripts\deploy.bat
-```
-
-### Option 2: Manual Deployment (Linux/Mac)
-
-```bash
-# Make script executable
-chmod +x scripts/deploy.sh
-
-# Run deployment
-bash scripts/deploy.sh
-```
-
-### Option 3: Manual Deployment (PowerShell)
+From your local Windows machine:

 ```powershell
-# Run the deployment script from project root
+# Deploy to home lab (beepc)
 .\scripts\deploy.ps1
 ```

-### Option 4: Automated Deployment (GitHub Actions)
+This will:
+1. Copy all project files to your home lab server
+2. Build the Docker image
+3. Create and start the container with persistent storage
+4. Set up systemd service for auto-start

-1. Add your SSH private key to GitHub Secrets:
-   - Go to your repository Settings > Secrets and variables > Actions
-   - Add a new secret named `SSH_PRIVATE_KEY`
-   - Paste your SSH private key content
+The app will be available at `http://homebase.sketchferret.com`

-2. Push to main branch:
-   ```bash
-   git add .
-   git commit -m "Initial commit"
-   git push origin main
-   ```
+### Persistent Data Storage

-3. The app will automatically deploy to spencer@beepc
+Images and database are stored in a Docker volume (`homebase-data`) that persists across container restarts and updates.
+
+For detailed deployment instructions, see [DOCKER_DEPLOYMENT.md](DOCKER_DEPLOYMENT.md)
+
+### Option 2: Manual Deployment
+
+Manual deployment steps are documented in [DOCKER_DEPLOYMENT.md](DOCKER_DEPLOYMENT.md#manual-updates)

 ## SSH Setup

@@ -114,28 +136,62 @@ sudo apt-get install docker-compose-plugin

 ## API Endpoints

- `GET /` - Main endpoint, returns app status
+### Status & Health
+- `GET /api/status` - App status and version
 - `GET /health` - Health check endpoint

+### Image Management
+- `GET /api/images` - List all images (with pagination & filtering)
+- `GET /api/images/{id}` - Get image details
+- `GET /api/images/{id}/download` - Download image file
+- `POST /api/images` - Fetch and save new image
+- `DELETE /api/images/{id}` - Delete image
+
+### Tagging
+- `POST /api/images/{id}/tags` - Add tags to image
+- `GET /api/tags` - List all available tags
+
+### Administration
+- `GET /api/stats` - Storage statistics
+- `POST /api/verify` - Verify all images for corruption
+- `POST /api/cleanup` - Clean up old images
+- `GET /api/fetcher/status` - Image fetcher status
+
+See [IMAGE_STORAGE_GUIDE.md](IMAGE_STORAGE_GUIDE.md) for detailed API documentation.
+
 ## Project Structure

 ```
 HomeBase/
 ├── scripts/
-│   ├── deploy.ps1           # PowerShell deployment script
-│   ├── deploy.sh            # Bash deployment script
-│   └── deploy.bat           # Windows batch deployment script
+│   ├── deploy.ps1                    # PowerShell deployment script
+│   ├── deploy.sh                     # Bash deployment script
+│   └── deploy.bat                    # Windows batch deployment script
+├── lib/
+│   ├── database.js                   # SQLite operations & schema
+│   ├── storage.js                    # File storage & checksums
+│   └── image-fetcher.js             # Image fetching service
+├── routes/
+│   └── images.js                     # Image API endpoints
+├── public/
+│   ├── index.html
+│   ├── app.js
+│   └── styles.css
 ├── .github/
 │   └── workflows/
-│       └── deploy.yml       # GitHub Actions workflow
-├── .dockerignore            # Docker ignore file
-├── .gitignore              # Git ignore file
-├── Dockerfile              # Docker configuration
-├── docker-compose.yml      # Docker Compose configuration
-├── homebase.service        # Systemd service file
-├── package.json            # Node.js dependencies
-├── server.js               # Express server
-└── README.md               # This file
+│       └── deploy.yml                # GitHub Actions workflow
+├── .dockerignore                     # Docker ignore file
+├── .gitignore                        # Git ignore file
+├── Dockerfile                        # Docker configuration
+├── docker-compose.yml                # Docker Compose with volumes
+├── homebase.service                  # Systemd service file
+├── package.json                      # Node.js dependencies
+├── server.js                         # Express server
+├── setup.js                          # Setup & CLI tool
+├── image-sources.json                # Image source configuration
+├── IMAGE_STORAGE_GUIDE.md           # Complete image storage docs
+├── DOCKER_DEPLOYMENT.md             # Docker deployment guide
+└── README.md                         # This file
 ```

 ## Deployment Process
@@ -212,16 +268,44 @@ ssh spencer@beepc "sudo systemctl restart homebase"
 ssh spencer@beepc "sudo systemctl disable homebase"
 ```

-## Environment Variables
+## Configuration

-You can customize settings by modifying [docker-compose.yml](docker-compose.yml):
+### Image Sources
+
+Edit `image-sources.json` to configure which image URLs to fetch from:
+
+```json
+{
+  "sources": [
+    {
+      "name": "Webcam Feed",
+      "url": "https://your-service.com/image.jpg",
+      "tags": ["webcam", "monitoring"],
+      "enabled": true
+    }
+  ],
+  "fetchInterval": 0.033
+}
+```
+
+**Fetch Intervals** - Edit `fetchInterval` (in minutes):
+- `0.0167` = 1 second
+- `0.033` = 2 seconds (default, recommended for steady updates)
+- `0.05` = 3 seconds
+- `1` = 1 minute
+- `2.5` = 2.5 minutes (original default)
+
+The fetcher will automatically start pulling images from enabled sources at this interval.
+
+### Environment Variables
+
+Customize settings in [docker-compose.yml](docker-compose.yml):

 ```yaml
 environment:
-  - PORT=3001  # Internal port (reverse proxy maps to standard HTTP)
-  - DOMAIN=homebase.sketchferret.com  # Your domain
-ports:
-  - "3001:3001"  # Port mapping for reverse proxy backend
+  - PORT=3001
+  - DOMAIN=homebase.sketchferret.com
+  - NODE_ENV=production
 ```

 ## Security Notes