Deployment and Infrastructure

Introduction

The Deployment and Infrastructure section provides comprehensive guidance for deploying, configuring, and maintaining the ESG Management System in production environments. Built on the robust Frappe framework, the system supports various deployment scenarios from single-server installations to enterprise-scale distributed deployments with high availability and scalability requirements.

1. System Requirements

Minimum Requirements

Hardware Requirements

• CPU: 2 cores (4 cores recommended)
• RAM: 4GB (8GB recommended for production)
• Storage: 20GB SSD (100GB+ recommended for production)
• Network: Stable internet connection for external integrations

Software Requirements

[project]
name = "esg_enviroment"
requires-python = ">=3.10"
dependencies = [
    # "frappe~=15.0.0" # Installed and managed by bench
]

[build-system]
requires = ["flit_core >=3.4,<4"]
build-backend = "flit_core.buildapi"

Core Dependencies

• Python: 3.10 or higher
• Frappe Framework: v14+ (v15+ recommended)
• MariaDB: 10.6+ or MySQL 8.0+
• Redis: 6.0+ for caching and session management
• Node.js: 16+ for frontend asset compilation
• Nginx: For reverse proxy and static file serving

Production Requirements

Recommended Production Specifications

• CPU: 8+ cores for high-throughput ESG data processing
• RAM: 16GB+ (32GB for large organizations)
• Storage: 500GB+ SSD with backup storage
• Network: High-bandwidth connection for IoT data ingestion
• Load Balancer: For multi-server deployments

2. Installation Methods

Method 1: Standard Frappe Bench Installation

Prerequisites Installation

# Install system dependencies (Ubuntu/Debian)
sudo apt update
sudo apt install -y python3-dev python3-pip python3-venv
sudo apt install -y mariadb-server mariadb-client
sudo apt install -y redis-server nginx
sudo apt install -y nodejs npm
sudo apt install -y git curl wget

# Install Frappe Bench
sudo pip3 install frappe-bench

Site Creation and App Installation

# Initialize bench
bench init frappe-bench --frappe-branch version-15
cd frappe-bench

# Create new site
bench new-site esg-site.localhost \
    --db-host localhost \
    --mariadb-root-password your_password \
    --admin-password admin_password

# Get ESG Environment app
bench get-app esg_enviroment https://github.com/

# Install app on site
bench --site esg-site.localhost install-app esg_enviroment

# Start development server
bench start

Method 2: Docker Deployment

Docker Compose Configuration

version: '3.8'

services:
  esg-app:
    image: frappe/frappe:v15
    environment:
      - FRAPPE_SITE_NAME=esg-site.localhost
      - DB_HOST=mariadb
      - REDIS_CACHE=redis-cache:6379
      - REDIS_QUEUE=redis-queue:6379
    volumes:
      - ./apps:/home/frappe/frappe-bench/apps
      - ./sites:/home/frappe/frappe-bench/sites
    depends_on:
      - mariadb
      - redis-cache
      - redis-queue

  mariadb:
    image: mariadb:10.6
    environment:
      - MYSQL_ROOT_PASSWORD=your_password
      - MYSQL_DATABASE=esg_database
    volumes:
      - mariadb_data:/var/lib/mysql

  redis-cache:
    image: redis:6-alpine
    command: redis-server --maxmemory 256mb --maxmemory-policy allkeys-lru

  redis-queue:
    image: redis:6-alpine

  nginx:
    image: nginx:alpine
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - ./nginx.conf:/etc/nginx/nginx.conf
      - ./ssl:/etc/nginx/ssl
    depends_on:
      - esg-app

volumes:
  mariadb_data:

Method 3: Cloud Deployment

AWS Deployment Architecture

3. Configuration Management

Application Configuration

Environment-Specific Settings

# site_config.json for production
{
    "db_name": "esg_production",
    "db_password": "secure_password",
    "redis_cache": "redis://redis-cache:6379",
    "redis_queue": "redis://redis-queue:6379",
    "redis_socketio": "redis://redis-socketio:6379",
    
    # ESG-specific configurations
    "enable_async_mqtt_processing": true,
    "mqtt_broker_url": "mqtt://iot-broker.company.com:1883",
    "esg_data_retention_days": 2555,  # 7 years
    "max_iot_batch_size": 1000,
    
    # Performance settings
    "background_workers": 4,
    "gunicorn_workers": 8,
    "socketio_workers": 2,
    
    # Security settings
    "encryption_key": "your-encryption-key",
    "session_timeout": 3600,
    "enable_two_factor_auth": true,
    
    # Backup settings
    "backup_frequency": "daily",
    "backup_retention": 30,
    "offsite_backup_enabled": true
}

ESG Constants Configuration

# ESG-specific constants and configurations
RENEWABLE_SOURCES = ["Wind", "Solar", "Hydro", "Biomass", "Ocean", "Waste-to-Energy", "Other Renewable"]
FOSSIL_SOURCES = ["Coal", "Oil", "Shale Oil", "Shale Gas", "Peat", "Other Fossil"]

# Default Emission Factors (kg CO2e per kWh)
EMISSION_FACTORS = {
    "Electricity": 0.5,  # Grid average, can be refined based on country/region
    "Gas": 0.2,
    "Coal": 0.9,
    "Oil": 0.7,
    "Wind": 0.01,
    "Solar": 0.02,
    "Hydro": 0.03
}

# ESG Period Types
PERIOD_TYPE_YEARLY = "Yearly"
PERIOD_TYPE_QUARTERLY = "Quarterly"
PERIOD_TYPE_MONTHLY = "Monthly"
PERIOD_TYPE_CUSTOM = "Custom"

Database Configuration

MariaDB Optimization for ESG Data

-- ESG-specific database optimizations
[mysqld]
# Memory settings for large ESG datasets
innodb_buffer_pool_size = 8G
innodb_log_file_size = 1G
innodb_flush_log_at_trx_commit = 2

# Query optimization
query_cache_size = 256M
query_cache_type = 1
tmp_table_size = 512M
max_heap_table_size = 512M

# Connection settings
max_connections = 200
max_user_connections = 150

# ESG-specific table optimizations
innodb_file_per_table = 1
innodb_compression_default = 1

# Backup settings
log_bin = mysql-bin
binlog_format = ROW
expire_logs_days = 7

Index Optimization for ESG Queries

-- Optimized indexes for ESG data queries
CREATE INDEX idx_esg_metric_value_period ON `tabESG Metric Value` (esg_period, metric_definition);
CREATE INDEX idx_realtime_data_facility_time ON `tabRealtime Data` (facility_id, timestamp);
CREATE INDEX idx_ghg_emissions_org_period ON `tabGHG Emissions` (organization, reporting_period);
CREATE INDEX idx_energy_consumption_facility_date ON `tabEnergy Consumption` (facility_id, reporting_period);
CREATE INDEX idx_water_consumption_org_date ON `tabWater Consumption` (organization, reporting_period);

-- Composite indexes for complex ESG reporting queries
CREATE INDEX idx_esg_report_metrics ON `tabESG Report` (organization, esg_period, standard);
CREATE INDEX idx_metric_value_verification ON `tabESG Metric Value` (verification_status, organization);

4. Security Configuration

SSL/TLS Configuration

Nginx SSL Configuration

server {
    listen 443 ssl http2;
    server_name esg.company.com;
    
    # SSL Configuration
    ssl_certificate /etc/nginx/ssl/esg.company.com.crt;
    ssl_certificate_key /etc/nginx/ssl/esg.company.com.key;
    ssl_protocols TLSv1.2 TLSv1.3;
    ssl_ciphers ECDHE-RSA-AES256-GCM-SHA512:DHE-RSA-AES256-GCM-SHA512;
    ssl_prefer_server_ciphers off;
    ssl_session_cache shared:SSL:10m;
    ssl_session_timeout 10m;
    
    # Security Headers
    add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;
    add_header X-Frame-Options DENY always;
    add_header X-Content-Type-Options nosniff always;
    add_header X-XSS-Protection "1; mode=block" always;
    add_header Referrer-Policy "strict-origin-when-cross-origin" always;
    
    # ESG Application Proxy
    location / {
        proxy_pass http://127.0.0.1:8000;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        
        # WebSocket support for real-time ESG data
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
    }
    
    # Static files for ESG dashboard
    location /assets {
        alias /home/frappe/frappe-bench/sites/assets;
        expires 1y;
        add_header Cache-Control "public, immutable";
    }
}

Authentication and Authorization

Multi-Factor Authentication Setup

# Enable MFA in site_config.json
{
    "enable_two_factor_auth": true,
    "two_factor_methods": ["SMS", "Email", "Authenticator App"],
    "mfa_required_roles": ["ESG Manager", "System Manager"],
    "session_timeout_with_mfa": 7200,
    "session_timeout_without_mfa": 1800
}

Role-Based Access Control

# ESG-specific role permissions
ESG_ROLES = {
    "ESG Manager": {
        "permissions": ["read", "write", "create", "delete", "export"],
        "doctypes": ["ESG Report", "ESG Metric Value", "ESG Goals", "ESG Organization"],
        "restrictions": []
    },
    "ESG User": {
        "permissions": ["read", "write", "create"],
        "doctypes": ["ESG Metric Value", "Realtime Data", "Energy Consumption"],
        "restrictions": ["cannot_delete_verified_data"]
    },
    "ESG Auditor": {
        "permissions": ["read", "export"],
        "doctypes": ["ESG Report", "Data Verification", "Third Party Audit"],
        "restrictions": ["read_only_access"]
    }
}

5. Performance Optimization

Application Performance

Caching Strategy

# Redis caching configuration for ESG data
CACHE_CONFIG = {
    "esg_metrics_cache": {
        "timeout": 3600,  # 1 hour
        "key_pattern": "esg:metrics:{organization}:{period}"
    },
    "dashboard_cache": {
        "timeout": 300,   # 5 minutes
        "key_pattern": "esg:dashboard:{user}:{facility}"
    },
    "report_cache": {
        "timeout": 7200,  # 2 hours
        "key_pattern": "esg:report:{report_id}:{version}"
    }
}

# Background job optimization
BACKGROUND_JOBS = {
    "esg_metric_calculation": {
        "queue": "long",
        "timeout": 1800,
        "retry": 3
    },
    "iot_data_processing": {
        "queue": "short",
        "timeout": 300,
        "retry": 2
    },
    "report_generation": {
        "queue": "long",
        "timeout": 3600,
        "retry": 1
    }
}

Database Performance Tuning

-- ESG-specific query optimizations
-- Partitioning for large ESG datasets
CREATE TABLE `tabRealtime Data Partitioned` (
    -- table structure
) PARTITION BY RANGE (YEAR(timestamp)) (
    PARTITION p2023 VALUES LESS THAN (2024),
    PARTITION p2024 VALUES LESS THAN (2025),
    PARTITION p2025 VALUES LESS THAN (2026),
    PARTITION p_future VALUES LESS THAN MAXVALUE
);

-- Archiving strategy for historical ESG data
CREATE EVENT esg_data_archival
ON SCHEDULE EVERY 1 MONTH
DO
BEGIN
    -- Archive data older than 7 years to separate table
    INSERT INTO `tabESG Metric Value Archive`
    SELECT * FROM `tabESG Metric Value`
    WHERE value_date < DATE_SUB(NOW(), INTERVAL 7 YEAR);
    
    DELETE FROM `tabESG Metric Value`
    WHERE value_date < DATE_SUB(NOW(), INTERVAL 7 YEAR);
END;

6. Monitoring and Logging

Application Monitoring

Health Check Endpoints

# ESG system health checks
@frappe.whitelist(allow_guest=True)
def health_check():
    """Comprehensive health check for ESG system"""
    health_status = {
        "status": "healthy",
        "timestamp": now_datetime(),
        "checks": {
            "database": check_database_connection(),
            "redis": check_redis_connection(),
            "mqtt_broker": check_mqtt_connection(),
            "background_jobs": check_background_jobs(),
            "disk_space": check_disk_space(),
            "memory_usage": check_memory_usage()
        }
    }
    
    # Determine overall status
    if any(check["status"] == "error" for check in health_status["checks"].values()):
        health_status["status"] = "unhealthy"
    elif any(check["status"] == "warning" for check in health_status["checks"].values()):
        health_status["status"] = "degraded"
    
    return health_status

Monitoring Dashboard Configuration

# Prometheus monitoring for ESG metrics
global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'esg-application'
    static_configs:
      - targets: ['localhost:8000']
    metrics_path: '/api/method/esg_enviroment.monitoring.metrics'
    
  - job_name: 'esg-database'
    static_configs:
      - targets: ['localhost:3306']
    
  - job_name: 'esg-redis'
    static_configs:
      - targets: ['localhost:6379']

# Grafana dashboard for ESG KPIs
dashboard_config:
  esg_metrics:
    - panel: "Real-time Data Ingestion Rate"
      query: "rate(esg_iot_data_received_total[5m])"
    - panel: "ESG Report Generation Time"
      query: "histogram_quantile(0.95, esg_report_generation_duration_seconds)"
    - panel: "Data Quality Score"
      query: "avg(esg_data_quality_score)"

Logging Configuration

Structured Logging for ESG Operations

# ESG-specific logging configuration
LOGGING_CONFIG = {
    "version": 1,
    "disable_existing_loggers": False,
    "formatters": {
        "esg_formatter": {
            "format": "%(asctime)s - %(name)s - %(levelname)s - %(message)s - %(esg_context)s"
        }
    },
    "handlers": {
        "esg_file": {
            "class": "logging.handlers.RotatingFileHandler",
            "filename": "/var/log/esg/esg_operations.log",
            "maxBytes": 100000000,  # 100MB
            "backupCount": 10,
            "formatter": "esg_formatter"
        },
        "esg_audit": {
            "class": "logging.handlers.RotatingFileHandler",
            "filename": "/var/log/esg/esg_audit.log",
            "maxBytes": 100000000,
            "backupCount": 50,  # Keep more audit logs
            "formatter": "esg_formatter"
        }
    },
    "loggers": {
        "esg_operations": {
            "handlers": ["esg_file"],
            "level": "INFO",
            "propagate": False
        },
        "esg_audit": {
            "handlers": ["esg_audit"],
            "level": "INFO",
            "propagate": False
        }
    }
}

7. Backup and Disaster Recovery

Backup Strategy

Automated Backup Configuration

#!/bin/bash
# ESG system backup script

# Configuration
BACKUP_DIR="/backup/esg"
RETENTION_DAYS=30
S3_BUCKET="esg-backups"
ENCRYPTION_KEY="/etc/esg/backup.key"

# Database backup
bench --site esg-site.localhost backup \
    --with-files \
    --compress \
    --backup-path $BACKUP_DIR

# Encrypt backup
gpg --cipher-algo AES256 \
    --compress-algo 1 \
    --symmetric \
    --passphrase-file $ENCRYPTION_KEY \
    --output $BACKUP_DIR/esg-backup-$(date +%Y%m%d).sql.gz.gpg \
    $BACKUP_DIR/esg-backup-$(date +%Y%m%d).sql.gz

# Upload to S3
aws s3 cp $BACKUP_DIR/esg-backup-$(date +%Y%m%d).sql.gz.gpg \
    s3://$S3_BUCKET/daily/

# Cleanup old backups
find $BACKUP_DIR -name "*.gz.gpg" -mtime +$RETENTION_DAYS -delete

# Verify backup integrity
bench --site esg-site.localhost backup-verification \
    --backup-file $BACKUP_DIR/esg-backup-$(date +%Y%m%d).sql.gz.gpg

Disaster Recovery Plan

Recovery Procedures

8. Scaling and Load Balancing

Horizontal Scaling

Multi-Server Deployment

# Docker Swarm configuration for ESG scaling
version: '3.8'

services:
  esg-app:
    image: esg/frappe:latest
    deploy:
      replicas: 3
      placement:
        constraints:
          - node.role == worker
      resources:
        limits:
          cpus: '2'
          memory: 4G
        reservations:
          cpus: '1'
          memory: 2G
    networks:
      - esg-network

  esg-worker:
    image: esg/frappe:latest
    command: bench worker --queue short,long
    deploy:
      replicas: 2
      placement:
        constraints:
          - node.role == worker
    networks:
      - esg-network

  nginx-lb:
    image: nginx:alpine
    ports:
      - "80:80"
      - "443:443"
    deploy:
      placement:
        constraints:
          - node.role == manager
    configs:
      - source: nginx_config
        target: /etc/nginx/nginx.conf
    networks:
      - esg-network

networks:
  esg-network:
    driver: overlay
    attachable: true

Load Balancing Configuration

Nginx Load Balancer for ESG Services

upstream esg_backend {
    least_conn;
    server esg-app-1:8000 weight=3 max_fails=3 fail_timeout=30s;
    server esg-app-2:8000 weight=3 max_fails=3 fail_timeout=30s;
    server esg-app-3:8000 weight=2 max_fails=3 fail_timeout=30s;
}

upstream esg_socketio {
    ip_hash;  # Sticky sessions for WebSocket
    server esg-app-1:9000;
    server esg-app-2:9000;
    server esg-app-3:9000;
}

server {
    listen 80;
    server_name esg.company.com;
    
    # Health check endpoint
    location /health {
        access_log off;
        proxy_pass http://esg_backend;
        proxy_set_header Host $host;
    }
    
    # WebSocket for real-time ESG data
    location /socket.io/ {
        proxy_pass http://esg_socketio;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_set_header Host $host;
        proxy_cache_bypass $http_upgrade;
    }
    
    # Main application
    location / {
        proxy_pass http://esg_backend;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        
        # Timeouts for long-running ESG reports
        proxy_connect_timeout 60s;
        proxy_send_timeout 300s;
        proxy_read_timeout 300s;
    }
}

Key Deployment Features

Installation Options

• Frappe Bench: Standard development and production deployment
• Docker: Containerized deployment with orchestration support
• Cloud: AWS, Azure, GCP deployment with managed services
• Hybrid: On-premises with cloud backup and disaster recovery

Security Features

• SSL/TLS: End-to-end encryption with modern cipher suites
• Authentication: Multi-factor authentication and role-based access
• Network Security: Firewall configuration and VPN support
• Data Protection: Encryption at rest and in transit

Performance Optimization

• Caching: Multi-level caching strategy for ESG data
• Database Tuning: Optimized for large ESG datasets
• Load Balancing: Horizontal scaling with intelligent load distribution
• Monitoring: Comprehensive monitoring and alerting

Conclusion

The Deployment and Infrastructure guide provides a comprehensive framework for successfully deploying and maintaining the ESG Management System in production environments. From basic single-server installations to enterprise-scale distributed deployments, the system is designed to meet the demanding requirements of modern ESG data management while ensuring security, performance, and reliability.

Related Documentation

• Backend Systems - Technical implementation details
• Architecture - Overall system architecture
• Customization - System customization and extensions
• Data Management - Data processing and management