DevOps & Developer Onboarding Guide¶

This guide is both a developer onboarding manual and a production DevOps reference for the PeerPay deployment, with a focus on the fabric-api and blockchain-explorer services.

For a refresher on how the components fit together, see the Architecture Overview.

1. Developer Onboarding¶

If you are new to the team, here is what you need to get started.

Prerequisites¶

• Docker (24+) and Docker Compose v2 (docker compose ...)
• Make (used by every repository's automation)
• Node.js v20 or later (only if you run the API or Explorer outside Docker)
• Git with SSH access to the PeerPay GitHub organisation
• Network access to the Hyperledger Fabric configuration (orderer, peers, CAs)

Local Development Setup¶

1. Clone the repository

   git clone <repo-url> deployment-github
   cd deployment-github

2. Create your environment file

   Copy .env.example to .env and fill in the required variables (e.g. JWT secrets, database credentials, Fabric network paths). Do not commit .env.

3. Bring up the Fabric network first

   The API and Explorer expect a running Fabric network. Bootstrap it from the peerpay-fabric repository:

   cd peerpay-fabric
   make setup        # or: make teardown-and-setup
   cd ..

4. Start the API and Explorer

   docker compose up -d --build

5. Verify

   Service	URL
   Blockchain Explorer	http://localhost:3000
   Fabric API	http://localhost:5000 or http://localhost:8080

---

2. Docker Container Management¶

This section covers the Docker Compose commands you will use most often. Where the file name docker-compose.yml is the default, the -f flag is optional.

Start Services¶

Start the fabric-api and blockchain-explorer services:

docker compose up -d --build fabric-api
docker compose up -d --build blockchain-explorer

Rebuild and Restart¶

When you change source code, dependencies, or the Dockerfile, rebuild and restart the affected service:

docker compose up -d --build <service_name>

Restart Services¶

Restart without rebuilding (useful for picking up .env changes):

docker compose restart blockchain-explorer
# Or restart everything:
docker compose restart

Stop Services¶

Stop a single service:

docker compose stop blockchain-explorer

Stop all services and remove containers and the default network:

docker compose down

Enter a Running Container¶

Useful for ad-hoc debugging:

docker exec -it blockchain-explorer /bin/bash
# If bash is not available in the image, fall back to sh:
docker exec -it blockchain-explorer /bin/sh

Restart Policy Options¶

In production, every long-running service in docker-compose.yml should declare a restart policy:

Policy	Description
no	Do not restart the container (default)
on-failure	Restart only if the container exits with a non-zero status
always	Always restart, even after a manual docker stop
unless-stopped	Restart unless explicitly stopped (recommended for production)

Healthcheck Example¶

Define a healthcheck so Docker (and orchestrators) know when a container is actually ready:

services:
  fabric-api:
    image: peerpay/fabric-api:latest
    restart: unless-stopped
    healthcheck:
      test: ["CMD", "curl", "-fsS", "http://localhost:5000/health"]
      interval: 30s
      timeout: 5s
      retries: 3
      start_period: 30s

---

3. Logging & Monitoring¶

Viewing Logs¶

Real-time logs for a specific service:

docker compose logs -f fabric-api

Tail the last 100 lines:

docker compose logs -f --tail=100 blockchain-explorer

Filter by timestamp:

docker compose logs --since 30m fabric-api

Production Monitoring¶

In production, do not rely solely on docker logs. Recommended setup:

1. Log aggregation — forward Docker logs to Grafana Loki (or another aggregator) using the loki Docker logging driver, or by running Promtail / Vector as a sidecar.
2. Healthchecks — every service in docker-compose.yml declares a healthcheck so unresponsive containers can be detected and restarted automatically.
3. Resource monitoring — use docker stats for ad-hoc inspection and Prometheus + Grafana for long-term metrics on CPU, memory, disk, and network.
4. Application metrics — the API exposes a /metrics endpoint where applicable; scrape it from Prometheus.
5. Alerting — configure Grafana alerts on container crash loops, elevated 5xx rates, and Fabric peer / orderer disconnects.

---

4. Configuration & Secrets¶

• Source of truth: every service reads its configuration from environment variables. A .env.example is checked in; the real .env is never committed.
• Local development: keep .env out of Git via .gitignore. Use a password manager (or 1Password / AWS Secrets Manager) to share secrets between developers.
• Production:
  • Inject secrets from a secrets manager at deploy time (AWS Secrets Manager, AWS SSM Parameter Store, or a Kubernetes Secret).
  • Do not bake secrets into Docker images.
  • Rotate JWT signing keys, database passwords and Kafka credentials on a fixed schedule.
• Fabric crypto material: certificates and private keys live in crypto-config/ and the wallet/ directory. Treat them like secrets; never commit them and never paste them into chat or tickets.

---

5. CI/CD Pipeline Strategy¶

Deployment is automated via GitHub Actions.

Standard CI Flow¶

1. Lint & Test — on every pull request to main, run ESLint, unit tests, and integration tests.
2. Build Docker images — once the PR is merged, build images for fabric-api and blockchain-explorer.
3. Tag images — tag each image with both the short Git SHA and a semantic version (e.g. peerpay/fabric-api:1.4.2, peerpay/fabric-api:abc1234).
4. Push to registry — push to the private container registry on AWS ECR.

Standard CD Flow¶

1. Deploy to staging — automatically deploy the new images to the staging environment for QA.
2. Smoke tests — run a small post-deploy test suite that hits /health, lists peers, and submits a known-safe transaction.
3. Manual approval — a release engineer approves the production deploy from the GitHub Actions UI.
4. Deploy to production — pull the same image tag that passed staging and roll it out (docker compose pull && docker compose up -d).
5. Post-deploy verification — re-run smoke tests against production and confirm dashboards are healthy.

---

6. Rollback & Recovery Strategy¶

In the event of a production failure, follow these steps to roll back.

Rollback by Image Tag (preferred)¶

If images are pulled from a registry, simply redeploy the previous tag:

# 1. Identify the last known-good tag (from the registry or release notes)
PREV_TAG=1.4.1

# 2. Pin the tag in the environment / compose file and redeploy
export PEERPAY_API_IMAGE=peerpay/fabric-api:${PREV_TAG}
docker compose pull fabric-api
docker compose up -d fabric-api

Rollback via Docker Compose (Local / Manual deployments)¶

If images are built directly on the server and a recent docker compose up -d --build broke the system:

1. Find the previous stable image with docker images.

2. Revert the Git repository to the previous stable commit:

   git checkout <previous-stable-commit-hash>

3. Rebuild and restart from the stable codebase:

   docker compose up -d --build

Database Rollback¶

Application data lives in PostgreSQL and (for ledger state) inside the Fabric peers themselves.

• For PostgreSQL, restore from the most recent automated snapshot (RDS) or pg_dump archive.
• The Fabric ledger is append-only; you cannot "roll back" committed blocks. If a bad chaincode invocation has been committed, the fix is a compensating transaction or a chaincode upgrade — not a rewind.
• After a rollback, always re-run the post-deploy smoke tests.

---

7. Troubleshooting¶

fabric-api cannot reach the network¶

• Check that peerpay-fabric is up: docker ps | grep peer0.
• Verify the connection profile path and the FABRIC_* environment variables in .env.
• Confirm TLS certificates have not expired: openssl x509 -enddate -noout -in <cert.pem>.

blockchain-explorer shows a blank page or "Failed to fetch"¶

• Confirm VITE_API_BASE_URL points to the correct Fabric API origin.
• Open the browser dev tools and check for CORS errors — the API must include the Explorer origin in its CORS allow-list.

Container restarts in a loop¶

• Tail the logs: docker compose logs --tail=200 <service>.
• Inspect the exit code: docker inspect <container> --format '{{.State.ExitCode}} {{.State.Error}}'.
• Common causes: missing .env value, port already in use, healthcheck failing during start_period.

Mermaid diagrams are not rendered in the docs site¶

• Make sure dependencies are installed: pip install -r requirements.txt.
• Confirm mkdocs.yml lists mermaid2 under plugins and that you are running the latest mkdocs serve.