Building a Data-Driven Gatsby Static Site Generation Pipeline with ActiveMQ and SciPy

We encountered a tricky monitoring requirement for a core trading system. The business side needed a near-real-time anomaly reporting dashboard for system performance metrics, but the operations team had strong reservations about introducing any dynamic web application that required continuous database polling or stateful backend services. The failure of any monitoring component could not, under any circumstances, impact the stability of the core system. While traditional dynamic dashboards like Grafana are powerful, their pull model would place significant strain on our time-series database, and their maintenance requires dedicated resources. What we needed was an extremely lightweight, highly available, and fully decoupled reporting solution.

After a few rounds of discussion, a slightly unconventional architecture emerged: building a message-driven, fully static reporting site. The core of the entire process is asynchronicity and pre-generation. After receiving metrics data, a backend service uses scientific computing libraries to perform anomaly detection, generates visual charts and data summaries, and then triggers a static site generator to rebuild the entire reporting site. End-users always access a pure HTML/CSS/JS site, requiring no backend database or API calls, which maximizes frontend reliability.

Technology Selection Decisions

The technology stack for this architecture was chosen to prioritize decoupling, reliability, and efficient offline processing capabilities.

1. Message Queue: ActiveMQ
   We chose ActiveMQ over more modern alternatives like Kafka or RabbitMQ, primarily for two reasons. First, our project environment already had a mature ActiveMQ cluster, and the operations team had in-depth knowledge of it. Second, we needed a reliable message broker that supported Durable Topics to ensure no metric messages were lost, even if the analysis consumer went down. ActiveMQ’s JMS durable subscription model perfectly met this requirement.

2. Data Processing & Visualization: Python, SciPy, Seaborn
   Python is the natural choice in the data analysis space. We wouldn’t introduce heavyweight frameworks like Spark or Flink, as the metric data volume was within the processing capacity of a single machine. SciPy provides a mature library of statistical functions, sufficient for implementing anomaly detection algorithms based on statistical models, such as Z-score-based outlier detection. Seaborn can then easily generate high-quality SVG or PNG chart files from the analysis results, which can be directly consumed by the frontend. We would build a lightweight consumer framework around this core functionality.

3. Static Site Generator: Gatsby
   We chose Gatsby for the frontend, over Next.js or Astro, and the key factor was its powerful ecosystem of data source plugins and its GraphQL-based data layer. Gatsby can easily use JSON files and images from the local filesystem as data sources. This means our Python processor only needs to write the analysis results (data summaries and chart files) to a predefined directory, and Gatsby can query this data via GraphQL at build time to render the pages.

Architectural Flow Overview

The lifecycle of the entire data pipeline is clear and avoids complex real-time interactions.

graph TD
    A[Core Trading System] -- gRPC/REST --> B(Metrics Producer);
    B -- STOMP/JMS --> C{ActiveMQ Topic: system.metrics};
    C -- Durable Subscription --> D[Python Consumer];
    subgraph D [Python Consumer Service]
        D1[STOMP Connection & Listener];
        D2[Data Window Management];
        D3[SciPy Anomaly Detection];
        D4[Seaborn Chart Generation];
        D5[Write to Filesystem];
        D6[Trigger Gatsby Build];
    end
    D1 --> D2 --> D3 --> D4 --> D5 --> D6;
    
    subgraph E [Gatsby Build Environment]
        E1[gatsby-source-filesystem];
        E2[GraphQL Data Layer];
        E3[React Component Rendering];
    end

    F[Filesystem /reports] -- Reads from --> E1;
    D5 -- Writes JSON/SVG --> F;

    G[Build Script build.sh] -- Executes gatsby build --> E;
    D6 -- Invokes --> G;

    E -- Generates Static Files --> H(Nginx/CDN);
    I[Ops / Business Users] -- Browser Access --> H;

The key to this flow is that the process from the Python consumer to the final static site is unidirectional and asynchronous. The frontend presentation is decoupled in time from the backend processing.

Step-by-Step Implementation

1. Configuring a Durable Topic in ActiveMQ

To ensure no messages are lost, we need to configure a durable topic in ActiveMQ. In conf/activemq.xml, make sure the persistenceAdapter is set up correctly, for example, using the default KahaDB.

<!-- conf/activemq.xml -->
<broker xmlns="http://activemq.apache.org/schema/core" brokerName="localhost" dataDirectory="${activemq.data}">

    <destinationPolicy>
        <policyMap>
            <policyEntries>
                <policyEntry topic=">" advisoryForConsumed="true">
                    <pendingMessageLimitStrategy>
                        <constantPendingMessageLimitStrategy limit="1000"/>
                    </pendingMessageLimitStrategy>
                </policyEntry>
            </policyEntries>
        </policyMap>
    </destinationPolicy>

    <managementContext>
        <managementContext createConnector="false"/>
    </managementContext>

    <persistenceAdapter>
        <kahaDB directory="${activemq.data}/kahadb"/>
    </persistenceAdapter>
    
    <!-- ... other configurations ... -->
</broker>

We don’t need any special configuration for a specific topic; the default persistence mechanism will suffice.

2. The Metrics Producer (Python Simulation)

A simple Python script simulates the core system continuously sending API response latency metrics. In a real project, this part would be embedded in the business logic or a separate monitoring agent. We use the stomp.py library to communicate with ActiveMQ.

# producer.py
import stomp
import time
import json
import random
import logging
from datetime import datetime

# Configure logging
logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s')

# ActiveMQ connection settings
HOSTS = [('localhost', 61613)]
TOPIC = '/topic/system.metrics'

class MetricProducer:
    def __init__(self, hosts):
        self.conn = stomp.Connection(host_and_ports=hosts)
        self.connected = False
        try:
            self.conn.connect('admin', 'admin', wait=True)
            self.connected = True
            logging.info("Successfully connected to ActiveMQ.")
        except stomp.exception.ConnectFailedException:
            logging.error("Could not connect to ActiveMQ. Please check if the service is running and credentials are correct.")

    def send_metric(self, metric_data):
        if not self.connected:
            logging.warning("Connection is down. Cannot send message.")
            return
        
        try:
            message = json.dumps(metric_data)
            self.conn.send(body=message, destination=TOPIC)
            logging.info(f"Successfully sent metric: {message}")
        except Exception as e:
            logging.error(f"An error occurred while sending a message: {e}")

    def disconnect(self):
        if self.connected:
            self.conn.disconnect()
            logging.info("Disconnected from ActiveMQ.")

def generate_metric():
    """Generates simulated API response time metrics, with occasional anomalies."""
    base_latency = random.uniform(50, 150)
    # About 5% chance of generating an abnormally high latency
    if random.random() < 0.05:
        latency = base_latency + random.uniform(200, 500)
    else:
        latency = base_latency
    
    return {
        'timestamp': datetime.utcnow().isoformat(),
        'metric_name': 'api_response_time_ms',
        'value': round(latency, 2),
        'source': 'payment_service'
    }

if __name__ == "__main__":
    producer = MetricProducer(HOSTS)
    if producer.connected:
        try:
            while True:
                metric = generate_metric()
                producer.send_metric(metric)
                time.sleep(random.uniform(0.5, 2))
        except KeyboardInterrupt:
            logging.info("Producer is shutting down.")
        finally:
            producer.disconnect()

This producer continuously sends JSON data containing a timestamp, metric name, and value.

3. The Core: Python Consumer and Analysis Framework

This is the heart of the system. It needs to robustly handle connections and messages, and integrate data analysis, visualization, and the build trigger logic. We’ll design it as a configurable class.

# consumer.py
import stomp
import json
import logging
import time
import os
import subprocess
from collections import deque
from datetime import datetime

import numpy as np
from scipy import stats
import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt

# --- Configuration Section ---
# ActiveMQ Settings
HOSTS = [('localhost', 61613)]
TOPIC = '/topic/system.metrics'
CLIENT_ID = 'gatsby-report-builder-1'
SUBSCRIPTION_NAME = 'durable-report-subscription'

# Data Processing Settings
WINDOW_SIZE = 100  # Analyze the last 100 data points
ANOMALY_THRESHOLD_Z_SCORE = 3.0  # Z-score threshold for anomalies

# Output Path Settings (Gatsby project will read from here)
GATSBY_PROJECT_PATH = '/path/to/your/gatsby-site'
OUTPUT_DATA_PATH = os.path.join(GATSBY_PROJECT_PATH, 'src/data')
OUTPUT_IMAGE_PATH = os.path.join(GATSBY_PROJECT_PATH, 'static/reports')
BUILD_SCRIPT_PATH = os.path.join(GATSBY_PROJECT_PATH, 'build.sh')

# Ensure output directories exist
os.makedirs(OUTPUT_DATA_PATH, exist_ok=True)
os.makedirs(OUTPUT_IMAGE_PATH, exist_ok=True)

# Logging Configuration
logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s')


class MetricConsumer(stomp.ConnectionListener):
    def __init__(self, conn):
        self.conn = conn
        self.data_window = deque(maxlen=WINDOW_SIZE)
        self.all_data = [] # Complete history for plotting

    def on_error(self, frame):
        logging.error(f'Received an error: {frame.body}')

    def on_disconnected(self):
        logging.warning('Disconnected from ActiveMQ. Attempting to reconnect in 5 seconds...')
        time.sleep(5)
        connect_and_subscribe(self.conn)

    def on_message(self, frame):
        try:
            msg = json.loads(frame.body)
            value = msg.get('value')
            timestamp_str = msg.get('timestamp')
            
            if value is None or timestamp_str is None:
                logging.warning(f"Received malformed message: {frame.body}")
                return

            timestamp = datetime.fromisoformat(timestamp_str)
            
            self.data_window.append(value)
            self.all_data.append({'timestamp': timestamp, 'value': value, 'is_anomaly': False})
            
            # Only analyze when the window is full
            if len(self.data_window) == WINDOW_SIZE:
                self.analyze_window()

        except json.JSONDecodeError:
            logging.error(f"Failed to parse message body: {frame.body}")
        except Exception as e:
            logging.error(f"An unknown error occurred while processing message: {e}")

    def analyze_window(self):
        """Use SciPy for anomaly detection."""
        window_array = np.array(self.data_window)
        z_scores = np.abs(stats.zscore(window_array))
        
        # Check if the last point in the window is an anomaly
        last_point_z_score = z_scores[-1]
        
        if last_point_z_score > ANOMALY_THRESHOLD_Z_SCORE:
            latest_data_point = self.all_data[-1]
            latest_data_point['is_anomaly'] = True
            
            logging.warning(
                f"Anomaly detected! "
                f"Value: {latest_data_point['value']:.2f}, "
                f"Z-score: {last_point_z_score:.2f} > {ANOMALY_THRESHOLD_Z_SCORE}"
            )
            
            # Generate the report
            self.generate_report()

    def generate_report(self):
        """Generate charts with Seaborn and trigger the Gatsby build."""
        logging.info("Generating anomaly report...")
        
        report_timestamp = datetime.now().strftime('%Y%m%d_%H%M%S')
        
        # 1. Prepare data
        df = pd.DataFrame(self.all_data)
        
        # 2. Plot with Seaborn
        plt.figure(figsize=(12, 6))
        sns.set_theme(style="whitegrid")
        plot = sns.lineplot(x='timestamp', y='value', data=df, label='API Response Time (ms)')
        
        # Highlight anomalies
        anomalies = df[df['is_anomaly']]
        if not anomalies.empty:
            sns.scatterplot(x='timestamp', y='value', data=anomalies, color='red', s=100, label='Anomaly Detected', zorder=5)

        plt.title('API Response Time Anomaly Report')
        plt.xlabel('Timestamp (UTC)')
        plt.ylabel('Response Time (ms)')
        plt.legend()
        plt.tight_layout()
        
        image_filename = f'anomaly_report_{report_timestamp}.svg'
        image_filepath = os.path.join(OUTPUT_IMAGE_PATH, image_filename)
        plt.savefig(image_filepath)
        plt.close()
        logging.info(f"Chart saved to: {image_filepath}")
        
        # 3. Generate JSON data file
        report_data = {
            'report_id': report_timestamp,
            'generated_at': datetime.utcnow().isoformat(),
            'anomaly_count': len(anomalies),
            'latest_anomaly': anomalies.iloc[-1].to_dict() if not anomalies.empty else None,
            'plot_path': f'/reports/{image_filename}', # Public path accessible by Gatsby
            'summary_stats': df['value'].describe().to_dict()
        }
        
        json_filepath = os.path.join(OUTPUT_DATA_PATH, 'latest_report.json')
        with open(json_filepath, 'w') as f:
            json.dump(report_data, f, indent=4)
        logging.info(f"Report data saved to: {json_filepath}")
        
        # 4. Trigger Gatsby build
        self.trigger_build()

    def trigger_build(self):
        logging.info("Triggering Gatsby site build...")
        try:
            # Ensure the build script is executable
            if not os.access(BUILD_SCRIPT_PATH, os.X_OK):
                os.chmod(BUILD_SCRIPT_PATH, 0o755)

            # Execute the build script in the Gatsby project directory
            # Use Popen instead of run to avoid blocking the consumer's main thread for too long
            process = subprocess.Popen(
                ['/bin/bash', BUILD_SCRIPT_PATH],
                stdout=subprocess.PIPE,
                stderr=subprocess.PIPE,
                cwd=GATSBY_PROJECT_PATH
            )
            logging.info(f"Build process started with PID: {process.pid}. Consumer will continue listening.")
            # In a real project, more sophisticated process management and logging would be needed.
            # stdout, stderr = process.communicate()
            # if process.returncode != 0:
            #     logging.error(f"Gatsby build failed:\n{stderr.decode()}")
            # else:
            #     logging.info(f"Gatsby build successful:\n{stdout.decode()}")
        except Exception as e:
            logging.error(f"An error occurred while executing the build script: {e}")


def connect_and_subscribe(conn):
    """Establishes a connection and sets up a durable subscription."""
    try:
        conn.connect('admin', 'admin', wait=True, headers={'client-id': CLIENT_ID})
        conn.subscribe(destination=TOPIC, id=1, ack='auto', headers={'subscription-type': 'topic', 'durable-subscription-name': SUBSCRIPTION_NAME})
        logging.info(f"Successfully connected and subscribed to topic '{TOPIC}' (Durable Subscription: {SUBSCRIPTION_NAME})")
    except stomp.exception.ConnectFailedException:
        logging.error("Connection failed, will retry on next cycle...")


if __name__ == "__main__":
    conn = stomp.Connection(host_and_ports=HOSTS)
    listener = MetricConsumer(conn)
    conn.set_listener('', listener)
    
    connect_and_subscribe(conn)
    
    # Keep the main thread alive
    while True:
        try:
            time.sleep(1)
        except KeyboardInterrupt:
            conn.disconnect()
            break

This consumer framework has several key design features:

• Durable Subscription: client-id and durable-subscription-name are crucial for implementing a durable subscription, ensuring that messages during consumer downtime are not lost.
• Sliding Window: collections.deque is used to implement an efficient fixed-size sliding window for real-time statistical calculations.
• Z-score Detection: scipy.stats.zscore is a simple yet powerful tool for identifying data points that deviate significantly from the mean. In a real project, this might be replaced with more complex algorithms like Isolation Forest or moving average crossovers.
• File Output: Analysis results are materialized as JSON and SVG files on disk. This is the core of the decoupling from Gatsby.
• Build Trigger: An external script is invoked via subprocess to trigger the build. This is a simple and direct approach.

4. Gatsby Frontend Integration

First, we need a simple build.sh script.

#!/bin/bash
# /path/to/your/gatsby-site/build.sh
echo "Starting static report site build..."
START_TIME=$SECONDS
# Clean up old artifacts and run the build
npm run clean && npm run build
ELAPSED_TIME=$(($SECONDS - $START_TIME))
echo "Build completed in $ELAPSED_TIME seconds."
# Here you could add commands to deploy to an Nginx directory or a CDN
# cp -r public/* /var/www/html/reports/

Next, configure Gatsby.

// gatsby-config.js
module.exports = {
  plugins: [
    // Read data from the filesystem
    {
      resolve: `gatsby-source-filesystem`,
      options: {
        name: `data`,
        path: `${__dirname}/src/data/`,
      },
    },
    // Transform JSON files into GraphQL nodes
    `gatsby-transformer-json`,
    // Other plugins...
    `gatsby-plugin-image`,
    `gatsby-plugin-sharp`,
    `gatsby-transformer-sharp`,
  ],
}

Finally, create a page to display the report.

// src/pages/index.js
import React from 'react';
import { graphql } from 'gatsby';

const ReportPage = ({ data }) => {
  const report = data.latestReportJson;

  if (!report) {
    return (
      <main style={{ padding: '2rem', fontFamily: 'sans-serif' }}>
        <h1>Anomaly Report Dashboard</h1>
        <p>No report generated yet. Waiting for data...</p>
      </main>
    );
  }

  return (
    <main style={{ padding: '2rem', fontFamily: 'sans-serif' }}>
      <h1>Anomaly Report Dashboard</h1>
      <p>Last Updated: {new Date(report.generated_at).toLocaleString()}</p>
      
      <div style={{ border: '1px solid #ccc', padding: '1rem', marginBottom: '2rem' }}>
        <h2>Latest Report: {report.report_id}</h2>
        <p><strong>Total Anomalies Detected in History:</strong> {report.anomaly_count}</p>
        
        <h3>Latest Anomaly Details:</h3>
        {report.latest_anomaly ? (
          <ul>
            <li>Timestamp: {new Date(report.latest_anomaly.timestamp).toLocaleString()}</li>
            <li>Value: {report.latest_anomaly.value} ms</li>
          </ul>
        ) : <p>None</p>}

        <h3>Summary Statistics (last {report.summary_stats.count} points):</h3>
        <ul>
          <li>Mean: {report.summary_stats.mean.toFixed(2)} ms</li>
          <li>Std Dev: {report.summary_stats.std.toFixed(2)} ms</li>
          <li>Min: {report.summary_stats.min.toFixed(2)} ms</li>
          <li>Max: {report.summary_stats.max.toFixed(2)} ms</li>
        </ul>
      </div>

      <div style={{ border: '1px solid #ccc', padding: '1rem' }}>
        <h2>Performance Trend Chart</h2>
        {/* SVGs can be loaded directly with an img tag */}
        <img src={report.plot_path} alt="Anomaly Report Chart" style={{ maxWidth: '100%', height: 'auto' }}/>
      </div>
    </main>
  );
};

export const query = graphql`
  query {
    latestReportJson {
      report_id
      generated_at
      anomaly_count
      plot_path
      latest_anomaly {
        timestamp
        value
      }
      summary_stats {
        count
        mean
        std
        min
        max
      }
    }
  }
`;

export default ReportPage;

Now, whenever the Python consumer detects an anomaly and generates the files, build.sh is triggered, Gatsby rebuilds the site, the contents of the public directory are updated, and after deployment, users will see the latest report.

Limitations and Future Iterations

While this architecture achieves the initial goals—a highly available, low-maintenance frontend and a fully decoupled backend—it has some obvious limitations.

First, reporting latency. The latency comes from two main sources: the time it takes to fill the data window and Gatsby’s build time. If anomalies occur frequently, the constant builds could become a system bottleneck or even lead to a queue of build tasks. This solution is not suitable for scenarios requiring second-level updates; it’s better suited for reporting cycles on the order of minutes or hours.

Second, consumer state management. The current consumer keeps the data window and historical data in memory. If the consumer process restarts, all state is lost, and the window must be refilled before analysis can begin again. An improvement would be to externalize this state, for instance, by storing it in Redis, allowing the consumer to restore its previous state upon restart.

Finally, the robustness of the build trigger mechanism. Directly calling a build script via subprocess is too simplistic. In a production environment, a more reliable approach would be for the consumer to call a webhook for a CI/CD platform (like Jenkins or GitLab CI). This would decouple the build process from the consumer and leverage the queuing, retry, and logging capabilities provided by the CI/CD platform, making the entire flow more robust. Additionally, debounce logic could be introduced to prevent too many build jobs from being triggered when multiple anomalies occur in a short period.