Streamlining Serverless Architecture Documentation with Diagram-as-Code

In the Justifai project, maintaining up-to-date and accurate architecture documentation for our evolving serverless environment was a consistent challenge. As features rapidly develop and AWS services are integrated, manual diagrams quickly become stale, leading to misunderstandings and increased onboarding time. This post outlines our journey to adopting a "diagram-as-code" approach to ensure our architectural blueprints are always current, versioned, and easily reproducible.

The Challenge of Evolving Architectures

Our Justifai application leverages a microservices-based serverless architecture, incorporating services like AWS Cognito for authentication, S3 for storage, DynamoDB with Global Secondary Indexes, AWS Textract for document processing, SQS/DLQ for message queuing, SNS for notifications, and CloudWatch for monitoring. While incredibly flexible, this dynamism meant that our previously ASCII-based architectural diagrams struggled to keep pace. Every change required manual updates, often leading to discrepancies between the documentation and the actual deployment.

Embracing Diagram-as-Code with Python's diagrams Library

To combat this, we turned to "diagram-as-code" using the Python diagrams library. This powerful tool allows us to define our cloud architecture using Python code, which then renders a visual diagram (typically PNG) via Graphviz. The benefits are immense:

• Version Control: Diagrams live alongside our codebase, versioned with Git.
• Reproducibility: Anyone can regenerate the diagram from the source code.
• Accuracy: Changes in the code (representing architectural updates) can directly trigger diagram updates.
• Automation: Integration into CI/CD pipelines is straightforward.

Prerequisites

To follow along, you'll need:

• Python 3.x
• The diagrams library: pip install diagrams
• Graphviz installed on your system (the dot binary must be accessible in your PATH).

Defining Your AWS Architecture in Python

Let's look at a simplified example of how we define a core part of our Justifai serverless workflow using diagrams. Imagine a process where documents are uploaded, trigger processing, and results are stored.

from diagrams import Diagram, Cluster
from diagrams.aws.compute import Lambda
from diagrams.aws.storage import S3
from diagrams.aws.database import Dynamodb
from diagrams.aws.analytics import Athena
from diagrams.aws.integration import SQS, SNS

with Diagram("Justifai Document Processing", show=False, direction="LR"):
    user_upload = S3("User Uploads")
    document_queue = SQS("Document Queue")
    processor_lambda = Lambda("Process Document")
    storage_table = Dynamodb("Document Metadata")
    notifier = SNS("Completion Topic")

    user_upload >> document_queue >> processor_lambda
    processor_lambda >> storage_table
    storage_table >> notifier

In this snippet, we import the necessary components from diagrams.aws and arrange them within a Diagram context. The >> operator defines the flow between services. This code is saved as docs/architecture.py.

Automating Diagram Generation and Documentation

With the Python script in place, generating the diagram is a simple command-line execution:

pip install diagrams
# Ensure Graphviz 'dot' command is installed and in your PATH
python docs/architecture.py

This command generates justifai_document_processing.png (or whatever you named your diagram). We then integrate this architecture.png into our project's README.md and architecture.md files, replacing outdated ASCII diagrams. We also ensure our internal documentation and external communications (like a LinkedIn post draft) reference the latest, most accurate visual.

Results

The shift to diagram-as-code has significantly improved the quality and maintainability of our Justifai project's documentation. We now have a single source of truth for our architecture, which is version-controlled, easily reproducible, and always reflects the current state of our serverless deployments. This reduces communication overhead, simplifies onboarding for new team members, and fosters a clearer understanding of our system's design.

Next Steps

Consider integrating the diagram generation into your CI/CD pipeline. This ensures that every pull request that modifies the architecture definition also updates the diagram, failing the build if the diagram generation process encounters issues or if the generated image isn't committed. This practice makes architectural documentation a first-class citizen in your development workflow.

Generated with Gitvlg.com