MaesterDiff: Because Comparing Maesters Just Got Twice as Fun!

As Microsoft 365 environments grow in complexity, maintaining security and compliance becomes increasingly challenging. Maester provides excellent testing capabilities for Microsoft 365 security compliance, but comparing results across different runs, tenants, or time periods often requires manual effort and disparate tools. This can lead to missed insights and inefficient audits.

I’ve faced these challenges firsthand, dealing with scattered JSON files and time-consuming comparisons. To address this, I’ve developed MaesterDiff: a PowerShell script that compares Maester test results and generates interactive HTML reports, making it easy to spot changes, trends, and anomalies in your security posture.

This tool is particularly useful for organizations that may not have access to premium tools like Inforcer or another similar tool, providing a free and effective alternative for comparing Maester results without the need for advanced infrastructure or licensing.

To utilize this script, ensure you have the following:

• PowerShell 7
• Maester JSON result files: Valid JSON files containing Maester test results with the expected structure (see File Structure section below).

This script is designed to provide comprehensive comparison capabilities for your Maester results, offering a variety of key features:

• Multiple Comparison Modes: Support for file-based, folder-based, cross-tenant, and timeline comparisons to suit different analysis needs.
• Interactive HTML Reports: Modern, responsive design with filtering, theming, export capabilities, and links to source reports for easy data exploration.
• Cross-Platform Support: Works seamlessly on Windows, macOS, and Linux environments with automatic OS detection and appropriate file handling.
• Flexible Filtering: Filter by tenant name, tenant ID, or limit the number of files processed.
• Summary Statistics: Optional display of summary metrics and progress indicators in the terminal.
• Export Functionality: Generate CSV exports directly from the HTML report, with cross-platform compatibility (exports to user’s Documents folder).
• Source Report Integration: Links to original Maester HTML reports for detailed drill-down.
• Theme Support: Light/dark theme switching for comfortable viewing.
• Differences-Only Mode: Focus on tests that have changed between comparisons.

This script is available in the Powershell Gallery. You can install it by the following line.
https://www.powershellgallery.com/packages/MaesterDiff/0.0.1

Before using MaesterDiff, you must first generate Maester reports. These reports are the JSON result files that the tool compares. To get started, follow the official Maester installation and usage instructions at https://maester.dev/docs/installation. This will guide you through setting up Maester, running tests, and producing the necessary JSON files for comparison.

Install-Script -Name MaesterDiff -Force

Install-Script -Name MaesterDiff -Force

File Comparison: This mode is ideal when you have specific result files you want to compare side-by-side. The script will process each file and compare all tests across them.
Folder Comparison: his mode automatically finds and compares the latest Maester result files in the specified folder. You can optionally filter by:

• TenantName: Compare only files from a specific tenant name.
• TenantId: Compare only files from a specific tenant ID.
• DifferencesOnly: Show only tests that have different results between files.
• MaxFiles: Limit the number of files to process (default: 10).

# File Comparison
.\MaesterDiff.ps1 -Files "TestResults-2025-06-01-120000.json", "TestResults-2025-06-02-120000.json"  

# Folder Comparison
.\MaesterDiff.ps1 -Folder "C:\MaesterResults" -TenantName "Contoso"

# File Comparison
.\MaesterDiff.ps1 -Files "TestResults-2025-06-01-120000.json", "TestResults-2025-06-02-120000.json"  

# Folder Comparison
.\MaesterDiff.ps1 -Folder "C:\MaesterResults" -TenantName "Contoso"

Compare the latest results between different tenants:

# Using tenant names
.\MaesterDiff.ps1 -Folder "C:\MaesterResults" -SourceTenantName "ContosoProd" -DestinationTenantName "ContosoTest"

# Using tenant IDs
.\MaesterDiff.ps1 -Folder "C:\MaesterResults" -SourceTenantId "12345678-1234-1234-1234-123456789012" -DestinationTenantId "87654321-4321-4321-4321-210987654321"

# Using tenant names
.\MaesterDiff.ps1 -Folder "C:\MaesterResults" -SourceTenantName "ContosoProd" -DestinationTenantName "ContosoTest"

# Using tenant IDs
.\MaesterDiff.ps1 -Folder "C:\MaesterResults" -SourceTenantId "12345678-1234-1234-1234-123456789012" -DestinationTenantId "87654321-4321-4321-4321-210987654321"

This mode finds the latest file for each specified tenant and compares their results. Perfect for comparing security compliance between production and test environments or different organizational units.

Enhance your comparisons with additional parameters:

• -ShowSummary: Display summary statistics and progress indicators in the console.
• -OutputPath: Specify the path for the generated HTML report (default: “TenantComparisonReport.html”).
• -Title: Customize the report title (default: “Microsoft 365 Tenant Comparison Report”).
• -OpenInBrowser: Automatically open the generated report in your default web browser.

.\MaesterDiff.ps1 -Folder "C:\MaesterResults" -SourceTenantName "ContosoProd" -DestinationTenantName "ContosoTest" -ShowSummary -OutputPath "MyComparison.html" -Title "Prod vs Test Comparison" -OpenInBrowser

.\MaesterDiff.ps1 -Folder "C:\MaesterResults" -SourceTenantName "ContosoProd" -DestinationTenantName "ContosoTest" -ShowSummary -OutputPath "MyComparison.html" -Title "Prod vs Test Comparison" -OpenInBrowser

The script generates an interactive HTML report featuring:

• Summary Statistics: Progress metrics and overview of changes between comparisons (total tests, matching/non-matching results for cross-tenant).
• Interactive Data Table: Filterable table with color-coded status indicators (Passed/Failed/Skipped/N/A) and progress indicators.
• Export Capabilities: Direct CSV export functionality from the browser, with cross-platform compatibility (automatically exports to user’s Documents folder).
• Links to Original Reports: Quick access to the source Maester HTML reports via buttons or dropdown.
• Theme Support: Light/dark theme switching for comfortable viewing.
• Responsive Design: Works across all modern web browsers and devices.
• Filtering Options: Filter by category, severity, tenant status, comparison type, and search terms.

The report includes source report integration, showing buttons or a dropdown to access detailed Maester reports for each compared file.

MaesterDiff simplifies what has traditionally been a complex task: comparing and analyzing Microsoft 365 security compliance test results across different scenarios. Whether you’re tracking changes over time, comparing tenant configurations, or auditing specific test runs, this tool provides clear, actionable insights to maintain your security posture.

The script’s cross-platform support, interactive reports, and flexible comparison modes make it an essential tool for IT administrators managing Microsoft 365 security compliance.