File Integrity Monitoring (FIM) is a critical process in maintaining the security and stability of computer systems. It involves monitoring and detecting changes to files and directories, ensuring that any unauthorized or unexpected modifications are promptly flagged. This is particularly important for protecting sensitive configurations, detecting malicious activity, and ensuring compliance with security standards.

Python offers an accessible and flexible way to implement FIM, thanks to its rich ecosystem of libraries and tools. With Python, we can efficiently track file changes by leveraging hashing algorithms and filesystem operations.

How File Integrity Monitoring Works

At its core, File Integrity Monitoring revolves around comparing the current state of a file or directory to its known baseline. Any deviation from the baseline may indicate unauthorized changes or potential issues.

Key Concepts of FIM:

1. File Attributes to Monitor:

• Size: Changes may indicate new data or tampering.
• Timestamps: Modifications to the “last modified” or “created” times.
• Hash Values: Cryptographic hashes (e.g., MD5, SHA256) uniquely identify file content.

2. Baseline vs. Current State:

• A baseline represents the initial state of files (e.g., their hash values).
• The current state is periodically captured and compared to the baseline.

3. Change Detection:

• File modifications, deletions, and additions are common types of changes flagged during monitoring.

The Role of Hashing in FIM

Hashing plays a critical role in ensuring that file content remains unchanged. A hash function generates a unique fixed-size string (a “fingerprint”) for any input data. Even a small alteration in the file will produce a completely different hash. Popular hashing algorithms include:

• MD5: Fast but less secure for cryptographic purposes.
• SHA256: More secure and widely used for integrity checks.

By comparing hashes, FIM can quickly and reliably detect if a file has been altered.

Prerequisites and Setup

Before diving into the implementation, you’ll need to ensure your environment is set up correctly.

Tools and Libraries Needed

• Python 3: Ensure Python is installed on your machine or use Google Colab for an online environment.
• hashlib: A built-in Python library for computing hashes.
• os: For interacting with the filesystem.
• time: (Optional) For periodic monitoring or time-based comparisons.

Setting Up in Colab

1. Open Google Colab in your browser.
2. Create a new notebook.
3. Import the necessary libraries (we’ll provide the code in the next sections).

By setting up in Colab, you can easily test the script without worrying about local configurations. Colab provides a seamless way to create, modify, and test files directly in its environment.

Implementation Steps

Step 1: Define the Scop

The first step in implementing File Integrity Monitoring is specifying which files or directories to monitor. This can be achieved using user input or by hardcoding the paths in your script.

For a basic example, the user should provide a directory that contains some files to monitor. If you’re using Google Colab, you can create and use a directory under the default Colab workspace (/content). Here’s how to set up an example directory in Colab:

Creating a Sample Directory in Colab

Run the following code in Colab to create a directory with some files for testing:

import os

# Create a sample directory and files
os.makedirs("/content/test_directory", exist_ok=True)
with open("/content/test_directory/file1.txt", "w") as f:
    f.write("This is file 1.")
with open("/content/test_directory/file2.txt", "w") as f:
    f.write("This is file 2.")
with open("/content/test_directory/file3.txt", "w") as f:
    f.write("This is file 3.")

print("Sample directory and files created at /content/test_directory")  

Example Directory Path to Provide

For the code snippet from Step 1, you can input the directory path:

/content/test_directory

This path contains the files you just created and can be used to test the functionality.

Step 2: Compute Baseline Hashes

In this step, we compute a baseline hash for each file in the specified directory. This baseline will serve as the reference to detect future changes.

We’ll use the hashlib library to calculate hash values (e.g., MD5 or SHA256) for file content. These hashes will be stored in a dictionary or exported for later use.

import hashlib

def compute_file_hash(file_path):
    """
    Compute the hash of a file using SHA256.
    """
    sha256_hash = hashlib.sha256()
    try:
        with open(file_path, "rb") as f:  # Open the file in binary read mode
            for byte_block in iter(lambda: f.read(4096), b""):  # Read in chunks
                sha256_hash.update(byte_block)
        return sha256_hash.hexdigest()
    except FileNotFoundError:
        return None  # Handle cases where the file doesn't exist

def create_baseline_hashes(file_paths):
    """
    Create a dictionary of file paths and their corresponding hash values.
    """
    baseline_hashes = {}
    for file_path in file_paths:
        file_hash = compute_file_hash(file_path)
        if file_hash:
            baseline_hashes[file_path] = file_hash
    return baseline_hashes

# Example usage
baseline_hashes = create_baseline_hashes(files)  # Use the file list from Step 1
print("Baseline hashes created:")
for file, hash_value in baseline_hashes.items():
    print(f"{file}: {hash_value}")

What This Does:

1. Compare Baseline and Current Hashes:
2. For each file in the baseline, compute the current hash and compare it to the baseline hash.
3. Detects modifications (hash mismatch) and deletions (None hash).

2. Detect New Files:

• Checks the current files in the directory.
• Identifies files that are not in the baseline and treats them as new.

3. Monitoring Interval:

• Uses time.sleep(interval) to periodically check the files.

Step 3: Monitor for Changes

In this step, we periodically check the files in the directory and compare their current hashes with the baseline hashes. This will help us detect changes such as file modifications, deletions, or new additions.

import time

def monitor_files(baseline_hashes, interval=10):
    """
    Monitor files for changes by comparing current hashes to the baseline.
    """
    print("Starting file monitoring...")
    while True:
        for file_path in baseline_hashes.keys():
            current_hash = compute_file_hash(file_path)
            if current_hash is None:
                print(f"File deleted: {file_path}")
            elif baseline_hashes[file_path] != current_hash:
                print(f"File modified: {file_path}")
        
        # Check for new files
        monitored_files = set(baseline_hashes.keys())
        current_files = set(get_files_to_monitor(directory_path))
        new_files = current_files - monitored_files
        for new_file in new_files:
            baseline_hashes[new_file] = compute_file_hash(new_file)
            print(f"New file detected: {new_file}")
        
        # Wait for the specified interval before the next check
        time.sleep(interval)

# Example usage
try:
    monitor_files(baseline_hashes, interval=15)  # Monitor every 15 seconds
except KeyboardInterrupt:
    print("Monitoring stopped.")

What This Does:

1. Compare Baseline and Current Hashes:
   1. For each file in the baseline, compute the current hash and compare it to the baseline hash.
   1. Detects modifications (hash mismatch) and deletions (None hash).
2. Detect New Files:
   1. Checks the current files in the directory.
   1. Identifies files that are not in the baseline and treats them as new.
3. Monitoring Interval:
   1. Uses time.sleep(interval) to periodically check the files.

Step 4: Logging and Notifications

In this step, we’ll enhance the monitoring script to log detected changes to a file and optionally send notifications (e.g., console logs or email). Logging is essential for tracking historical changes and debugging issues.

import logging

# Setup logging
def setup_logging(log_file="file_monitor.log"):
    """
    Configure the logging system to output changes to a log file.
    """
    logging.basicConfig(
        filename=log_file,
        filemode="a",  # Append mode
        format="%(asctime)s - %(levelname)s - %(message)s",
        level=logging.INFO,
    )
    logging.info("File Integrity Monitoring started.")

def monitor_files_with_logging(baseline_hashes, interval=10):
    """
    Monitor files for changes and log the detected events.
    """
    print("Starting file monitoring with logging...")
    while True:
        try:
            for file_path in baseline_hashes.keys():
                current_hash = compute_file_hash(file_path)
                if current_hash is None:
                    logging.warning(f"File deleted: {file_path}")
                    print(f"File deleted: {file_path}")
                elif baseline_hashes[file_path] != current_hash:
                    logging.warning(f"File modified: {file_path}")
                    print(f"File modified: {file_path}")

            # Check for new files
            monitored_files = set(baseline_hashes.keys())
            current_files = set(get_files_to_monitor(directory_path))
            new_files = current_files - monitored_files
            for new_file in new_files:
                baseline_hashes[new_file] = compute_file_hash(new_file)
                logging.info(f"New file detected: {new_file}")
                print(f"New file detected: {new_file}")

            # Wait before the next iteration
            time.sleep(interval)

        except KeyboardInterrupt:
            print("Monitoring stopped.")
            logging.info("File Integrity Monitoring stopped.")
            break

# Example usage
setup_logging()  # Initialize logging
try:
    monitor_files_with_logging(baseline_hashes, interval=15)  # Monitor every 15 seconds
except Exception as e:
    logging.error(f"An error occurred: {e}")  

What This Does:

1. Logging Configuration:
2. Logs events to a file (file_monitor.log) using the logging module.
3. Includes timestamps, event levels (INFO, WARNING, etc.), and event messages.

2. Log Types:

• INFO: For new files detected.
• WARNING: For modified or deleted files.
• ERROR: For exceptions.

3. Console Output:

• Changes are printed to the console as well for real-time observation.

Testing in Colab:

1. Run the Script:
2. Start monitoring with logging enabled.

2. Perform Changes:

• Modify, delete, or add files in /content/test_directory.

3. Check the Log File:

• View the log file (file_monitor.log) using:

with open("file_monitor.log", "r") as log:
    print(log.read())  

• The log will show events like:

2025-01-26 12:00:00,123 - WARNING - File modified: /content/test_directory/file1.txt
2025-01-26 12:00:10,456 - INFO - New file detected: /content/test_directory/new_file.txt

Optional Notification Enhancements:

1. Email Alerts: Use libraries like smtplib to send email notifications for critical events.
2. Real-Time Dashboards: Integrate with external monitoring tools like Slack or webhook-based platforms.

Wrapping up…

File Integrity Monitoring (FIM) is a crucial component of maintaining system security and ensuring the integrity of sensitive files. By systematically defining the scope of monitored files, computing baseline hashes, monitoring for changes, and logging events, organizations and individuals can proactively detect and respond to unauthorized modifications, deletions, or additions.

In this implementation, we demonstrated how Python can be used to build a lightweight yet effective FIM solution. Through incremental steps, including hashing algorithms and real-time monitoring, this approach provides a foundation for more advanced features such as automated notifications, integration with external security systems, or even cross-platform deployment.

This simple yet powerful script can be extended and tailored to fit specific use cases, from personal file tracking to enterprise-level security operations. By adopting such tools, you strengthen your defense against potential threats and ensure the continued trustworthiness of your systems.