# Clonit Agent

A CLI tool for snapshotting, sanitizing, sharing, and subsetting databases. PostgreSQL first, with MySQL planned.

## Features

- **Build** snapshots using native database tools (pg_dump directory format)
- **Load** snapshots or raw SQL files into destination databases
- **Push/Pull** snapshots to/from S3 and Cloudflare R2 with retry, checksums, and resume
- **Sanitize** snapshots through a full pipeline (restore -> transform -> dump)
- **Analyze** databases with AI to identify sensitive columns and auto-generate sanitization queries
- **Encrypted at rest** - sensitive fields (URLs, credentials) use AES-GCM encryption in SQLite
- **Web UI** - built-in browser interface for managing targets, snapshots, and storage profiles
- **CI/CD friendly** - headless operation, env var config, appropriate exit codes

## Standalone-First Design

Clonit is fully functional without any cloud connection. All core operations
work locally:

- **Build/Load** snapshots using local database tools
- **Push/Pull** to your own S3 or Cloudflare R2 using locally-configured storage profiles
- **Sanitize** data through local or Docker-based pipelines
- **Analyze** schemas with AI (uses your own Anthropic API key)

**Clonit Cloud** is an optional extension that adds:
- Centralized snapshot catalog accessible from any device
- Snapshot sharing with team members
- Organization management and SSO
- Managed storage via presigned URLs (no local S3 credentials needed)

## Quick Start

### Prerequisites

- Go 1.25+
- [Bun](https://bun.sh/) (for building the frontend)
- PostgreSQL client tools (`pg_dump`, `pg_restore`, `psql`) in PATH

### Build

```bash
make build
```

This builds the React frontend and compiles everything into a single binary at `bin/clonit`.

> **Repo layout:** `clonit-cli` is part of the `clonit` family alongside the private
> `clonit-cloud` and `clonit-shared` repos. It depends on
> `github.com/joescharf/clonit-shared` via a `replace => ../clonit-shared` directive,
> so check out `clonit-shared` as a sibling directory and set
> `GOPRIVATE=github.com/joescharf/*`.

### Initialize

```bash
clonit config init
```

Creates:
- `~/.config/clonit/config.yaml` - configuration with a generated secret key
- `~/.config/clonit/clonit.db` - SQLite metadata database

### Add a Target

```bash
clonit targets add \
  --name mydb \
  --src-url "postgres://user:pass@srchost:5432/mydb" \
  --dst-url "postgres://user:pass@localhost:5432/mydb_dev"
```

### Show Target Details

```bash
clonit targets show mydb
```

### Update a Target

Update individual fields of an existing target. Only flags you provide are changed.

```bash
clonit targets update <name> [flags]
```

Flags:

| Flag | Type | Notes |
|------|------|-------|
| `--name` | string | New target name |
| `--db-type` | string | Database type (postgres, postgresql; mysql planned) |
| `--src-url` | string | Source database URL |
| `--dst-url` | string | Destination database URL (empty to clear) |
| `--sanitize-dst-url` | string | Sanitize destination database URL (empty to clear) |
| `--sanitize-query-file` | string | Path to sanitization SQL query file (empty to clear) |
| `--storage-profile` | string | Storage profile name (empty to clear) |
| `--sanitized-storage-profile` | string | Storage profile for sanitized snapshots (empty to clear) |
| `--dump-jobs` | int | Number of parallel dump jobs |
| `--restore-jobs` | int | Number of parallel restore jobs |
| `--no-verify` | bool | Skip URL connectivity verification |

Examples:

```bash
# Update dump parallelism
clonit targets update mydb --dump-jobs 8

# Change destination URL
clonit targets update mydb --dst-url "postgres://user:pass@host:5432/dst"

# Clear optional field
clonit targets update mydb --dst-url ""

# Rename a target
clonit targets update mydb --name newname

# Skip URL verification
clonit targets update mydb --src-url "postgres://u:p@h:5432/db" --no-verify
```

### Build a Snapshot

```bash
clonit build mydb
```

If the target has a storage profile linked, the snapshot is automatically pushed to cloud storage after the build. Use `--no-push` to skip this:

```bash
clonit build mydb --no-push
```

### Load a Snapshot

```bash
# Load the latest snapshot
clonit load mydb

# Load a specific snapshot by index
clonit load mydb 0

# Load a SQL file
clonit load mydb /path/to/file.sql
```

### List Targets and Snapshots

```bash
clonit targets
clonit snapshots mydb
```

### Web UI

Start the built-in web interface:

```bash
clonit serve
```

This starts a web server at `http://127.0.0.1:8484` with a dashboard for managing targets, snapshots, and storage profiles. The UI is embedded in the binary — no separate frontend server needed.

```bash
# Custom host/port
clonit serve --host 0.0.0.0 --port 9090
```

## Commands

| Command | Description |
|---------|-------------|
| `config init` | Initialize config file and database |
| `serve` | Start the web UI server (default: `127.0.0.1:8484`) |
| `version` | Print version information |
| `login` | Log in to Clonit Cloud via your browser (OAuth device grant) |
| `logout` | Log out of Clonit Cloud and clear the stored session |
| `whoami` | Show the current Clonit Cloud login |
| `targets` | List all configured targets |
| `targets add` | Add a new target |
| `targets show <name>` | Show full details of a target |
| `targets update <name>` | Update an existing target's fields |
| `targets remove <name>` | Remove a target |
| `targets verify <name>` | Verify database connectivity for a target's URLs |
| `snapshots <target>` | List snapshots for a target |
| `build <target>` | Create a snapshot of the source database |
| `load <target> [index\|file.sql]` | Load a snapshot into the destination database |
| `push <target> [index]` | Push a snapshot to remote storage |
| `pull <target> [index]` | Pull a snapshot from remote storage |
| `sanitize <target> [index]` | Run the sanitization pipeline |
| `ephemeral <target> [index]` | Run sanitization in a disposable Docker container |
| `storage-profile` | List storage profiles |
| `storage-profile add` | Add a new storage profile |
| `storage-profile remove <name>` | Remove a storage profile |

## Clonit Cloud (Optional)

Clonit Cloud is an optional SaaS extension that adds team collaboration, cross-device access, and snapshot sharing. All core operations work without it.

### Two Push/Pull Pathways

**Local storage profile** (default):
```bash
clonit push mydb          # Uses target's linked storage profile → your S3/R2
clonit pull mydb          # Downloads from your S3/R2
```

**Clonit Cloud** (requires cloud config):
```bash
clonit push --cloud mydb  # Uploads via cloud presigned URL → org's managed storage
clonit pull --cloud mydb --snapshot-id <id>  # Downloads via cloud presigned URL
```

### Cloud Setup

```bash
# Configure cloud connection
clonit config edit         # Set cloud.url and cloud.api_key

# Register this agent with the cloud
clonit cloud register
```

The web UI includes a Cloud section for viewing shared snapshots, managing targets, and organization settings.

### Interactive Login

`clonit login` signs you in to Clonit Cloud through your browser using an OAuth 2.0
Device Authorization Grant (RFC 8628) against the central auth service — no API key
or password is typed into the terminal:

```bash
clonit login            # prints a code + URL, opens your browser, waits for approval
clonit whoami           # show the current login (and validate the session)
clonit logout           # clear the stored session
```

Only `cloud.url` is required: clonit discovers the auth service and client ID from
the cloud's `GET /api/v1/auth/config` endpoint. The session (access + refresh tokens)
is stored in `~/.config/clonit/credentials.json` (mode `0600`) and is refreshed
automatically. Flags: `--no-browser` (just print the URL), `--auth-url` /
`--client-id` (pin the auth service instead of discovering it), `--scope`.

> Interactive login is for humans. CI/CD and other headless automation should keep
> using a Clonit Cloud **API key** (`cloud.api_key`).

## Architecture

```
clonit-agent/
├── cmd/                          # Cobra CLI commands (thin layer)
├── internal/
│   ├── app/                      # App struct (dependency container)
│   ├── cloud/                    # Optional: Cloud API client
│   ├── config/                   # Config struct + Viper defaults
│   ├── database/                 # SQLite + goose migrations
│   │   └── migrations/           # SQL migration files
│   ├── docker/                   # Docker container management
│   ├── encryption/               # AES-GCM encrypt/decrypt
│   ├── engine/                   # DatabaseTools interface + factory
│   │   └── pgengine/             # PostgreSQL: pg_dump, pg_restore, psql
│   ├── introspect/               # Database schema introspection
│   ├── llm/                      # LLM client (Claude AI analysis)
│   ├── model/                    # Data models + repos
│   ├── output/                   # CLI output service (colored status, tables, quiet mode)
│   ├── service/                  # Business logic (build, load, push, pull, sanitize, analyze)
│   ├── storage/                  # Cloud storage (S3, R2) + transfer manager
│   ├── util/                     # File helpers, checksums, URL parsing
│   └── web/                      # HTTP server, chi router, SPA serving
│       └── handlers/             # API handlers
├── ui/                           # React frontend (Bun + shadcn/ui + TanStack Query)
│   └── src/
│       ├── components/           # Shared UI components
│       ├── lib/                  # API client
│       ├── pages/                # Page components (Dashboard, Targets, Storage Profiles)
│       └── types/                # TypeScript type definitions
├── cloud/                        # Clonit Cloud (multi-tenant SaaS)
├── main.go
├── Makefile
└── go.mod
```

### Key Design Decisions

- **SQLite for metadata** instead of all-in-YAML config. Targets, snapshots, and storage profiles live in `~/.config/clonit/clonit.db`.
- **YAML for settings only**. The config file holds secret key, working directory, log level, defaults - not target definitions.
- **Transparent encryption**. Sensitive fields are encrypted on write and decrypted on read using the `$aes$` prefix convention. The encryption key comes from `secret_key` in the config file.
- **Dependency injection** via an `App` struct created in `PersistentPreRunE` and passed to all commands.
- **DatabaseTools interface** allows swapping in different database engines (PostgreSQL now, MySQL later) or mocks for testing.
- **Transfer manager** handles retry with exponential backoff + jitter, SHA-256 checksum verification, and transient error detection for cloud storage operations.
- **Embedded web UI**. The React frontend is compiled by Bun and embedded into the Go binary via `//go:embed`. The `serve` command starts a chi-based HTTP server that serves the SPA and a REST API (`/api/v1/`). Sensitive fields (URLs, keys) are masked in all API responses.

## Configuration

### Config File (`~/.config/clonit/config.yaml`)

```yaml
secret_key: <auto-generated-hex-key>
working_directory: ~/.clonit
env: production
log:
  level: info    # debug, info, warn, error
  format: text   # text, json
docker:
  images:
    postgres: postgres:latest
    mysql: mysql:8.0-oracle
server:
  host: 127.0.0.1
  port: 8484
defaults:
  dump_jobs: 4
  restore_jobs: 4
  auto_push: true

# Optional: Clonit Cloud connection
cloud:
  url: ""         # Clonit Cloud server URL
  api_key: ""     # Organization API key (machine/CI credential)
  agent_id: ""    # Auto-set after 'clonit cloud register'
  auth_url: ""    # Optional: pin the auth service for 'clonit login' (else discovered)
  client_id: ""   # Optional: pin the OAuth client_id for 'clonit login' (else discovered)
```

Interactive `clonit login` sessions are stored separately in
`~/.config/clonit/credentials.json` (mode `0600`), not in `config.yaml`.

### Environment Variables

All config keys can be set via environment variables with the `CLONIT_` prefix. Nested keys use `__` as separator:

```bash
CLONIT_SECRET_KEY=mykey
CLONIT_LOG__LEVEL=debug
CLONIT_DEFAULTS__DUMP_JOBS=8
```

### Global Flags

```
--config string   Path to config file (default ~/.config/clonit/config.yaml)
--debug           Enable debug logging
--quiet           Suppress non-essential output
```

## Cloud Storage

### S3

```bash
clonit storage-profile add \
  --name my-s3 \
  --provider s3 \
  --bucket my-snapshots \
  --region us-east-1 \
  --access-key AKIA... \
  --secret-key wJalr...
```

### Cloudflare R2

```bash
clonit storage-profile add \
  --name my-r2 \
  --provider r2 \
  --bucket my-snapshots \
  --account-id <cloudflare-account-id> \
  --access-key <r2-access-key> \
  --secret-key <r2-secret-key>
```

### Linking a Target to Storage

```bash
clonit targets add \
  --name mydb \
  --src-url "postgres://..." \
  --storage-profile my-s3
```

You can also set a separate storage profile for sanitized snapshots, enabling
secure sharing of sanitized data without exposing originals:

```bash
clonit targets update mydb \
  --sanitized-storage-profile shared-s3
```

Then push/pull:

```bash
clonit push mydb      # Upload latest snapshot (uses correct profile based on type)
clonit pull mydb      # Download latest snapshot (defaults to sanitized if profile set)
clonit pull mydb --type original   # Explicitly pull original snapshot
```

## Sanitization

Sanitization runs a SQL transformation on a snapshot:

1. Restore the source snapshot to a sanitize destination database
2. Execute the sanitization SQL query
3. Dump the sanitized database to a new snapshot

### Setup

```bash
clonit targets add \
  --name mydb \
  --src-url "postgres://user:pass@prod:5432/mydb" \
  --dst-url "postgres://user:pass@localhost:5432/mydb_dev" \
  --sanitize-dst-url "postgres://user:pass@localhost:5432/mydb_sanitize" \
  --sanitize-query-file /path/to/sanitize.sql
```

### Example sanitize.sql

```sql
UPDATE users SET email = concat('user', id, '@example.com');
UPDATE users SET password_hash = '$2a$10$fake_hash';
TRUNCATE TABLE audit_logs;
```

### Run

```bash
clonit sanitize mydb
```

## Ephemeral Sanitization

Ephemeral sanitization runs the sanitization pipeline in a disposable Docker container, eliminating the need for a pre-configured `--sanitize-dst-url` database. Requires Docker to be running.

```bash
clonit ephemeral mydb              # Use latest snapshot + active query
clonit ephemeral mydb 0            # Use specific snapshot index
clonit ephemeral mydb --fast       # Skip LLM debug analysis on failure
clonit ephemeral mydb --build      # Force building a new snapshot first
clonit ephemeral mydb --image postgres:15  # Override Docker image
```

The pipeline orchestrates 8 steps: resolve snapshot, start container, wait for readiness, load snapshot, sanitize, debug analysis (on failure), dump sanitized snapshot, and stop container.

The web UI at `/targets/{id}` shows real-time step progress via SSE and maintains a history of ephemeral runs with retry support.

### Docker Configuration

```yaml
# ~/.config/clonit/config.yaml
docker:
  images:
    postgres: "postgres:16"
  ready_timeout: 60   # seconds to wait for container readiness
  password: "clonit"  # container database password
```

## Seed File

You can bootstrap targets and storage profiles from a YAML seed file (`clonit-seed.yaml`):

```yaml
storage_profiles:
  my-s3:
    provider: s3
    access_key: "AKIA..."
    secret_key: "secret..."
    region: us-east-1
    bucket: my-snapshots

targets:
  mydb:
    db_type: postgres
    src_url: "postgres://user:pass@host/db"
    dst_url: "postgres://user:pass@localhost/db_dev"
    storage_profile: my-s3
    dump_jobs: 8
```

Records are only imported if they don't already exist (idempotent).

## Development

### Build & Test

```bash
make build              # Build binary to bin/clonit
make test               # Run unit tests (short mode)
make test-integration   # Run all tests including integration
make lint               # Run golangci-lint
make clean              # Remove build artifacts
```

### Running Tests

```bash
go test -count=1 ./...                      # All tests
go test -count=1 -v ./internal/model/...    # Model tests (verbose)
go test -count=1 -v ./internal/service/...  # Service tests (verbose)
```

- **Model tests** use in-memory SQLite (`:memory:`)
- **Service tests** use mock `DatabaseTools` interface + in-memory SQLite
- **Storage tests** use mock `Storage` interface
- **Web handler tests** use in-memory SQLite with `httptest.NewRecorder`
- **Integration test** (`internal/web/integration_test.go`) runs a full CRUD round-trip through the API

### Frontend Development

For frontend development with hot module reloading, run the Go backend and Bun dev server simultaneously:

```bash
# Terminal 1: Start the Go backend
go run . serve

# Terminal 2: Start the Bun dev server (proxies /api/* to Go backend)
cd ui && bun --hot src/index.ts
```

The Bun dev server runs at `http://localhost:3000` and proxies all `/api/*` requests to the Go backend at `http://localhost:8484`.

### Build with Version Info

```bash
go build -ldflags "-X main.version=1.0.0 -X main.commit=$(git rev-parse --short HEAD) -X main.date=$(date -u +%Y-%m-%dT%H:%M:%SZ)" -o bin/clonit .
```

### Releasing & Distribution

Releases are built **locally** — macOS codesigning and notarization need the signing
certs in your Keychain and can't run in CI. The source repos are private; public
downloads are served from Cloudflare R2 (bucket `clonit-public`), the Homebrew tap,
and ghcr — **not** from GitHub Release assets.

`make release` always routes through 1Password so the release secrets are never
forgotten: `release` → `release-op` → `op run --env-file=release.env -- make release-exec` → goreleaser.

**One-time prerequisites:**

- Apple Developer signing certs in the Keychain, a `notarytool` keychain profile named
  `SCHARFNADO_LLC`, and `pycodesign.py` at `~/.local/bin/`.
- 1Password CLI (`op`), signed in. `release.env` is committed and holds **only** `op://`
  references — to the R2 API token (Object Read & Write on `clonit-public`) and the
  Homebrew tap GitHub token. No secrets live in the repo.

**Cut a release:**

```bash
git tag v0.2.0          # goreleaser releases off the latest tag
make release-snapshot   # optional dry run — builds, no signing/publish
make release            # 1Password → sign + notarize → R2 → Homebrew cask → ghcr
```

goreleaser signs + notarizes the macOS universal binary, uploads the archives to
`releases/<version>/` and `releases/latest/` on R2, publishes the Homebrew cask
(pointing at the R2 public URL), and pushes `ghcr.io/joescharf/clonit`. Users then
install with `brew install joescharf/tap/clonit` or download from
`https://pub-816a8822a6ce404ab522aa73c8f47274.r2.dev/releases/latest/<asset>`.

### Documentation

The user docs are an Astro Starlight site in `docs-user/`:

```bash
make docs-serve   # local dev server
make docs-build   # static build to docs-user/dist
```

They deploy to GitHub Pages via `.github/workflows/docs.yml` and are served at
`https://docs.clonit.ai`.

### Dependencies

| Package | Purpose |
|---------|---------|
| `github.com/spf13/cobra` | CLI framework |
| `github.com/spf13/viper` | Configuration management |
| `modernc.org/sqlite` | Pure Go SQLite driver (no CGO) |
| `github.com/pressly/goose/v3` | SQL migration management |
| `github.com/google/uuid` | UUID generation |
| `github.com/stretchr/testify` | Test assertions |
| `github.com/aws/aws-sdk-go-v2` | S3/R2 cloud storage |
| `github.com/go-chi/chi/v5` | HTTP router for web API |
| `github.com/go-chi/cors` | CORS middleware |
| `github.com/fatih/color` | Colored terminal output |
| `github.com/olekukonko/tablewriter` | Table formatting |
| `github.com/anthropics/anthropic-sdk-go` | Anthropic Claude API client for AI analysis |

### Frontend

| Package | Purpose |
|---------|---------|
| `react` / `react-dom` | UI framework |
| `react-router-dom` | Client-side routing |
| `@tanstack/react-query` | Server state management and caching |
| `shadcn/ui` + `tailwindcss` | Component library and styling |
| `lucide-react` | Icons |

## Security

- **Encryption at rest**: All sensitive fields (database URLs, API keys, storage credentials) are encrypted using AES-GCM with the `$aes$` prefix convention. The encryption key is stored in the config file.
- **Masked API responses**: The web UI API never returns plaintext credentials. URLs and keys are masked in all responses.
- **ReadHeaderTimeout**: The HTTP server sets `ReadHeaderTimeout` to prevent Slowloris-style attacks.
- **S3 key paths**: Uses `path.Join` (not `filepath.Join`) for S3 object keys to ensure forward-slash separators on all platforms.
- **Database connection lifecycle**: The SQLite connection is properly closed via `PersistentPostRunE` when the CLI exits.

## License

Copyright (c) 2026 Joe Scharf
