Run Multiple Python Scripts with subprocess

on
Python subprocess: Buffered vs Unbuffered with Error Handling

Python subprocess: Buffered vs Unbuffered with Error Handling



When running multiple Python scripts from a single runner, you often want to control how output is handled and whether one failure should stop the rest. This post shows how to:

  • Run multiple scripts via subprocess.Popen
  • Get real-time output
  • Skip failed scripts and continue execution

1. Buffered Mode (Default, No Real-Time Output)

Code Snippet


import subprocess

files = [
    r"C:\Users\pc\OneDrive\Desktop\subprocess\script1.py",
    r"C:\Users\pc\OneDrive\Desktop\subprocess\script2.py"
]

python_path = "python"

for file in files:
    print(f"Running {file}..\n{'='*30}")
    process = subprocess.Popen(
        ["python", file],
        stdout=subprocess.PIPE,
        stderr=subprocess.STDOUT,
        text=True,
        bufsize=1
    )
    for line in process.stdout:
        print(line, end='')
    process.wait()
    print(f"\n{'='*30}\nDone running {file}\n")

Explanation

  • subprocess.Popen runs the scripts one by one
  • Output is buffered — displayed only after script completes
  • Standard output and errors are merged for visibility

2. Unbuffered Mode (Real-Time Output)

Code Snippet


import subprocess

files = [
    r"C:\Users\pc\OneDrive\Desktop\subprocess\script1.py",
    r"C:\Users\pc\OneDrive\Desktop\subprocess\script2.py"
]

python_path = "python"

for file in files:
    print(f"Running {file}..\n{'='*30}")

    process = subprocess.Popen(
        ["python", "-u", file],  # Real-time output using -u
        stdout=subprocess.PIPE,
        stderr=subprocess.STDOUT,
        text=True,
        bufsize=1
    )
    for line in process.stdout:
        print(line, end='')
    process.wait()
    print(f"\n{'='*30}\nDone running {file}\n")

Explanation

  • -u switch forces unbuffered output
  • Use flush=True in child script print() for full real-time visibility
  • Great for monitoring long-running scripts

3. Unbuffered Mode + Skip on Error

Code Snippet


import subprocess

files = [
    r"C:\Users\pc\OneDrive\Desktop\subprocess\script1.py",
    r"C:\Users\pc\OneDrive\Desktop\subprocess\script2.py"
]

python_path = "python"

for file in files:
    print(f"Running {file}..\n{'='*30}")
    
    try:
        process = subprocess.Popen(
            ["python", "-u", file],
            stdout=subprocess.PIPE,
            stderr=subprocess.STDOUT,
            text=True,
            bufsize=1
        )

        for line in process.stdout:
            print(line, end='')

        exit_code = process.wait()

        if exit_code != 0:
            print(f"\n❌ Script failed with exit code {exit_code}: {file}\n")
        else:
            print(f"\n✅ Done running {file}\n")

    except Exception as e:
        print(f"\n❗ Exception while running {file}: {e}\n")

    print("="*30 + "\n")

What It Does

  • Wraps execution in try-except to catch and log failures
  • Continues to next script even if one fails
  • Outputs ✅ or ❌ based on exit code

Notes & Tips

  • Use print("text", flush=True) to force output visibility from child scripts
  • Run only automated scripts — avoid those needing user input

Tags

#Python #subprocess #errorhandling #bufferedoutput #realTimeOutput #automation

“In the world of code, Python is the language of simplicity, where logic meets creativity, and every line brings us closer to our goals.”— Only Python

📌 Follow Us And Stay Updated For Daily Updates

Comments

Post a Comment