Run Multiple Python Scripts with subprocess

• 📢 Stay updated with tutorials, projects & coding hacks — Join our WhatsApp Channel

Python subprocess: Buffered vs Unbuffered with Error Handling

• 📢 Stay updated with tutorials, projects & coding hacks — Join our WhatsApp Channel

When running multiple Python scripts from a single runner, you often want to control how output is handled and whether one failure should stop the rest. This post shows how to:

• Run multiple scripts via subprocess.Popen
• Get real-time output
• Skip failed scripts and continue execution

1. Buffered Mode (Default, No Real-Time Output)

Code Snippet

import subprocess

files = [
    r"C:\Users\pc\OneDrive\Desktop\subprocess\script1.py",
    r"C:\Users\pc\OneDrive\Desktop\subprocess\script2.py"
]

python_path = "python"

for file in files:
    print(f"Running {file}..\n{'='*30}")
    process = subprocess.Popen(
        ["python", file],
        stdout=subprocess.PIPE,
        stderr=subprocess.STDOUT,
        text=True,
        bufsize=1
    )
    for line in process.stdout:
        print(line, end='')
    process.wait()
    print(f"\n{'='*30}\nDone running {file}\n")
    

Explanation

• subprocess.Popen runs the scripts one by one
• Output is buffered — displayed only after script completes
• Standard output and errors are merged for visibility

2. Unbuffered Mode (Real-Time Output)

Code Snippet

import subprocess

files = [
    r"C:\Users\pc\OneDrive\Desktop\subprocess\script1.py",
    r"C:\Users\pc\OneDrive\Desktop\subprocess\script2.py"
]

python_path = "python"

for file in files:
    print(f"Running {file}..\n{'='*30}")

    process = subprocess.Popen(
        ["python", "-u", file],  # Real-time output using -u
        stdout=subprocess.PIPE,
        stderr=subprocess.STDOUT,
        text=True,
        bufsize=1
    )
    for line in process.stdout:
        print(line, end='')
    process.wait()
    print(f"\n{'='*30}\nDone running {file}\n")
    

Explanation

• -u switch forces unbuffered output
• Use flush=True in child script print() for full real-time visibility
• Great for monitoring long-running scripts

3. Unbuffered Mode + Skip on Error

Code Snippet

import subprocess

files = [
    r"C:\Users\pc\OneDrive\Desktop\subprocess\script1.py",
    r"C:\Users\pc\OneDrive\Desktop\subprocess\script2.py"
]

python_path = "python"

for file in files:
    print(f"Running {file}..\n{'='*30}")
    
    try:
        process = subprocess.Popen(
            ["python", "-u", file],
            stdout=subprocess.PIPE,
            stderr=subprocess.STDOUT,
            text=True,
            bufsize=1
        )

        for line in process.stdout:
            print(line, end='')

        exit_code = process.wait()

        if exit_code != 0:
            print(f"\n❌ Script failed with exit code {exit_code}: {file}\n")
        else:
            print(f"\n✅ Done running {file}\n")

    except Exception as e:
        print(f"\n❗ Exception while running {file}: {e}\n")

    print("="*30 + "\n")
    

What It Does

• Wraps execution in try-except to catch and log failures
• Continues to next script even if one fails
• Outputs ✅ or ❌ based on exit code

Notes & Tips

• Use print("text", flush=True) to force output visibility from child scripts
• Run only automated scripts — avoid those needing user input

Tags

#Python #subprocess #errorhandling #bufferedoutput #realTimeOutput #automation