HomeBase/README.md

# HomeBase

Self-hosted Node.js Docker application with automated SSH deployment and **long-term image storage** with corruption detection and tagging for ML training.

## Features

- 🚀 Express.js server
- 🐳 Docker containerized with persistent volumes
- 🔄 Automated deployment via SSH
- 📸 **Long-term image storage** with SQLite database
- 🏷️ **Tag-based organization** for machine learning datasets
- 🛡️ **Corruption detection** using SHA256 checksums
- 🔍 **RESTful API** for image management
- 🧪 **Automatic fetching** every 2-3 minutes from web services
- ✅ Health check endpoints
- 🔐 SSH key-based authentication

## Prerequisites

On your local machine:
- Node.js (for local testing)
- Docker (for local testing)
- SSH access to beepc server

On the remote server (beepc):
- Docker installed
- Docker Compose installed
- SSH access configured for user `spencer`

## Quick Start

### Local Development

```bash
# Install dependencies
npm install

# Initialize the image storage system
node setup.js init

# Configure image sources (edit image-sources.json)
# Set "enabled": true for sources you want to fetch from

# Run locally
npm start

# Visit http://localhost:3001
```

### Image Storage Features

Once running, manage images via the REST API:

```bash
# List all images
curl http://localhost:3001/api/images

# Add an image manually
curl -X POST http://localhost:3001/api/images \
  -H "Content-Type: application/json" \
  -d '{"source_url":"https://example.com/image.jpg","tags":["test"]}'

# Get storage statistics
curl http://localhost:3001/api/stats

# Verify all images for corruption
curl -X POST http://localhost:3001/api/verify
```

📖 **Full documentation:**
- [📸 Image Storage Guide](IMAGE_STORAGE_GUIDE.md) - Complete API reference, configuration, and usage
- [🐳 Docker Deployment Guide](DOCKER_DEPLOYMENT.md) - Deployment to home lab with persistent storage

### Deployment

### Quick Deployment to Home Lab (PowerShell)

From your local Windows machine:

```powershell
# Deploy to home lab (beepc)
.\scripts\deploy.ps1
```

This will:
1. Copy all project files to your home lab server
2. Build the Docker image
3. Create and start the container with persistent storage
4. Set up systemd service for auto-start

The app will be available at `http://homebase.sketchferret.com`

### Persistent Data Storage

Images and database are stored in a Docker volume (`homebase-data`) that persists across container restarts and updates.

For detailed deployment instructions, see [DOCKER_DEPLOYMENT.md](DOCKER_DEPLOYMENT.md)

### Option 2: Manual Deployment

Manual deployment steps are documented in [DOCKER_DEPLOYMENT.md](DOCKER_DEPLOYMENT.md#manual-updates)

## SSH Setup

Ensure you have SSH key-based authentication set up:

```bash
# Generate SSH key if you don't have one
ssh-keygen -t ed25519 -C "your_email@example.com"

# Copy key to remote server
ssh-copy-id spencer@beepc

# Test connection
ssh spencer@beepc
```

## Remote Server Setup

On beepc, ensure Docker is installed:

```bash
# Install Docker (Ubuntu/Debian)
curl -fsSL https://get.docker.com -o get-docker.sh
sudo sh get-docker.sh

# Add user to docker group
sudo usermod -aG docker spencer

# Install Docker Compose
sudo apt-get update
sudo apt-get install docker-compose-plugin

# Logout and login again for group changes to take effect
```

## API Endpoints

### Status & Health
- `GET /api/status` - App status and version
- `GET /health` - Health check endpoint

### Image Management
- `GET /api/images` - List all images (with pagination & filtering)
- `GET /api/images/{id}` - Get image details
- `GET /api/images/{id}/download` - Download image file
- `POST /api/images` - Fetch and save new image
- `DELETE /api/images/{id}` - Delete image

### Tagging
- `POST /api/images/{id}/tags` - Add tags to image
- `GET /api/tags` - List all available tags

### Administration
- `GET /api/stats` - Storage statistics
- `POST /api/verify` - Verify all images for corruption
- `POST /api/cleanup` - Clean up old images
- `GET /api/fetcher/status` - Image fetcher status

See [IMAGE_STORAGE_GUIDE.md](IMAGE_STORAGE_GUIDE.md) for detailed API documentation.

## Project Structure

```
HomeBase/
├── scripts/
│   ├── deploy.ps1                    # PowerShell deployment script
│   ├── deploy.sh                     # Bash deployment script
│   └── deploy.bat                    # Windows batch deployment script
├── lib/
│   ├── database.js                   # SQLite operations & schema
│   ├── storage.js                    # File storage & checksums
│   └── image-fetcher.js             # Image fetching service
├── routes/
│   └── images.js                     # Image API endpoints
├── public/
│   ├── index.html
│   ├── app.js
│   └── styles.css
├── .github/
│   └── workflows/
│       └── deploy.yml                # GitHub Actions workflow
├── .dockerignore                     # Docker ignore file
├── .gitignore                        # Git ignore file
├── Dockerfile                        # Docker configuration
├── docker-compose.yml                # Docker Compose with volumes
├── homebase.service                  # Systemd service file
├── package.json                      # Node.js dependencies
├── server.js                         # Express server
├── setup.js                          # Setup & CLI tool
├── image-sources.json                # Image source configuration
├── IMAGE_STORAGE_GUIDE.md           # Complete image storage docs
├── DOCKER_DEPLOYMENT.md             # Docker deployment guide
└── README.md                         # This file
```

## Deployment Process

The deployment script does the following:

1. Creates remote directory if it doesn't exist
2. Syncs files to remote server via rsync/scp
3. Stops existing Docker container
4. Builds new Docker image
5. Starts new container
6. Verifies container is running

## Troubleshooting

### Check logs on remote server

```bash
ssh spencer@beepc
cd /home/spencer/homebase
docker compose logs -f
```

### Check container status

```bash
ssh spencer@beepc "docker ps"
```

### Restart container

```bash
ssh spencer@beepc "cd /home/spencer/homebase && docker compose restart"
```

### Remove and rebuild

```bash
ssh spencer@beepc "cd /home/spencer/homebase && docker compose down && docker compose up -d --build"
```

## Accessing the App

Once deployed, access the app at:
- `http://homebase.sketchferret.com` (reverse proxied, no port needed)
- `http://beepc:3001` (direct access from local network)
- `http://localhost:3001` (direct access on the server)

**Note:** The app runs on port 3001 internally but is accessed via reverse proxy at the standard HTTP port.

## Auto-Start on Boot

The deployment script automatically sets up a systemd service that:
- Starts the Docker container on server boot
- Restarts the container if it crashes
- Runs as the spencer user

To manually manage the service:

```bash
# Check status
ssh spencer@beepc "sudo systemctl status homebase"

# Stop service
ssh spencer@beepc "sudo systemctl stop homebase"

# Start service
ssh spencer@beepc "sudo systemctl start homebase"

# Restart service
ssh spencer@beepc "sudo systemctl restart homebase"

# Disable auto-start
ssh spencer@beepc "sudo systemctl disable homebase"
```

## Configuration

### Image Sources

Edit `image-sources.json` to configure which image URLs to fetch from:

```json
{
  "sources": [
    {
      "name": "Webcam Feed",
      "url": "https://your-service.com/image.jpg",
      "tags": ["webcam", "monitoring"],
      "enabled": true
    }
  ],
  "fetchInterval": 0.033
}
```

**Fetch Intervals** - Edit `fetchInterval` (in minutes):
- `0.0167` = 1 second
- `0.033` = 2 seconds (default, recommended for steady updates)
- `0.05` = 3 seconds
- `1` = 1 minute
- `2.5` = 2.5 minutes (original default)

The fetcher will automatically start pulling images from enabled sources at this interval.

### Environment Variables

Customize settings in [docker-compose.yml](docker-compose.yml):

```yaml
environment:
  - PORT=3001
  - DOMAIN=homebase.sketchferret.com
  - NODE_ENV=production
```

## Security Notes

- Uses SSH key authentication (no passwords)
- Container runs as non-root user
- Only production dependencies installed in Docker image
- Health check endpoint for monitoring

## License

MIT