Understanding Error Handling and Exceptions in Python

When you write code, things don't always go as planned. A user might enter invalid data, a network connection might fail, or a file you're trying to read might not exist. Without proper error handling, these situations cause your program to crash with a confusing error message. Python's exception handling system gives you the power to anticipate these problems and respond intelligently.

An exception is an event that disrupts the normal flow of program execution. When Python encounters an error, it "raises" (or "throws") an exception. If you don't handle this exception, your program terminates. But with proper exception handling, you can catch these errors, deal with them gracefully, and keep your program running.

Built-in Exceptions in Python

Python comes with a rich hierarchy of built-in exception types, each designed to represent different kinds of errors. Understanding these exceptions helps you write more precise error-handling code and debug issues faster.

Common Built-in Exception Types

Let's see these exceptions in action:

# ValueError example
try:
    age = int("twenty-five")  # Can't convert text to integer
except ValueError as e:
    print(f"ValueError occurred: {e}")
    # Output: ValueError occurred: invalid literal for int() with base 10: 'twenty-five'

# TypeError example
try:
    result = "Hello" + 42  # Can't add string and integer
except TypeError as e:
    print(f"TypeError occurred: {e}")
    # Output: TypeError occurred: can only concatenate str (not "int") to str

# KeyError example
user_data = {"name": "Alice", "email": "alice@example.com"}
try:
    phone = user_data["phone"]  # Key doesn't exist
except KeyError as e:
    print(f"KeyError occurred: {e}")
    # Output: KeyError occurred: 'phone'

# IndexError example
numbers = [1, 2, 3, 4, 5]
try:
    value = numbers[10]  # Index out of range
except IndexError as e:
    print(f"IndexError occurred: {e}")
    # Output: IndexError occurred: list index out of range

# ZeroDivisionError example
try:
    result = 100 / 0
except ZeroDivisionError as e:
    print(f"ZeroDivisionError occurred: {e}")
    # Output: ZeroDivisionError occurred: division by zero

The Exception Hierarchy

All exceptions in Python inherit from the BaseException class. Understanding this hierarchy helps you catch exceptions at the right level of specificity:

BaseException
├── SystemExit
├── KeyboardInterrupt
├── GeneratorExit
└── Exception
    ├── StopIteration
    ├── ArithmeticError
    │   ├── ZeroDivisionError
    │   ├── OverflowError
    │   └── FloatingPointError
    ├── LookupError
    │   ├── IndexError
    │   └── KeyError
    ├── ValueError
    ├── TypeError
    ├── OSError
    │   ├── FileNotFoundError
    │   ├── PermissionError
    │   └── ConnectionError
    └── Many others...

⚠️ Important Note: You should generally catch Exception and its subclasses, not BaseException. Catching BaseException can interfere with system-level operations like KeyboardInterrupt (Ctrl+C) and SystemExit.

Try-Except-Else-Finally Blocks

Python's error handling revolves around the try-except-else-finally structure. Think of it as a safety net that catches errors before they crash your program. Each component has a specific purpose:

The Try Block

The try block contains code that might raise an exception. Python attempts to execute this code, and if an error occurs, it immediately stops and looks for a matching except block.

try:
    # Code that might raise an exception
    result = 10 / 2
    print(f"Result: {result}")

The Except Block

The except block catches and handles specific exceptions. You can have multiple except blocks to handle different error types differently.

# Basic except block
try:
    number = int(input("Enter a number: "))
    result = 100 / number
except ZeroDivisionError:
    print("You can't divide by zero!")
except ValueError:
    print("That's not a valid number!")

# Catching multiple exceptions in one block
try:
    data = {"name": "Bob"}
    age = data["age"]
except (KeyError, TypeError) as e:
    print(f"Data access error: {e}")

# Generic except (use cautiously)
try:
    risky_operation()
except Exception as e:
    print(f"Something went wrong: {e}")

💡 Best Practice: Always catch specific exceptions when possible. Using a bare except: (without specifying an exception type) is generally discouraged because it catches everything, including system exits and keyboard interrupts, making debugging difficult.

The Else Block

The else block runs only if no exception was raised in the try block. This is useful for code that should only execute when everything went smoothly.

def read_file_safely(filename):
    try:
        with open(filename, 'r') as file:
            content = file.read()
    except FileNotFoundError:
        print(f"Error: {filename} not found")
        return None
    except PermissionError:
        print(f"Error: No permission to read {filename}")
        return None
    else:
        # This runs only if no exception occurred
        print(f"Successfully read {filename}")
        return content
    
# Usage
content = read_file_safely("data.txt")
if content:
    print(f"File content length: {len(content)} characters")

The Finally Block

The finally block always executes, regardless of whether an exception occurred or not. This is perfect for cleanup operations like closing files, releasing locks, or closing network connections.

def process_database_transaction():
    connection = None
    try:
        connection = connect_to_database()
        execute_query(connection, "INSERT INTO users VALUES (...)")
        print("Transaction completed")
    except ConnectionError:
        print("Database connection failed")
    except QueryError:
        print("Query execution failed")
    else:
        print("Transaction successful, committing changes")
        connection.commit()
    finally:
        # This ALWAYS runs, even if there was a return statement
        if connection:
            connection.close()
            print("Database connection closed")

Complete Try-Except-Else-Finally Example

def divide_numbers(a, b):
    """Demonstrates all four blocks working together"""
    result = None
    
    try:
        print(f"Attempting to divide {a} by {b}")
        result = a / b
    except ZeroDivisionError:
        print("Error: Cannot divide by zero")
        return None
    except TypeError:
        print("Error: Both arguments must be numbers")
        return None
    else:
        # Only runs if no exception occurred
        print(f"Division successful: {a} / {b} = {result}")
    finally:
        # Always runs, regardless of exceptions
        print("Division operation completed")
    
    return result

# Test cases
print("=== Test 1: Normal division ===")
divide_numbers(10, 2)
# Output:
# Attempting to divide 10 by 2
# Division successful: 10 / 2 = 5.0
# Division operation completed

print("\n=== Test 2: Division by zero ===")
divide_numbers(10, 0)
# Output:
# Attempting to divide 10 by 0
# Error: Cannot divide by zero
# Division operation completed

print("\n=== Test 3: Invalid type ===")
divide_numbers(10, "two")
# Output:
# Attempting to divide 10 by two
# Error: Both arguments must be numbers
# Division operation completed

Custom Exceptions

While Python's built-in exceptions cover many scenarios, sometimes you need exceptions specific to your application's domain. Custom exceptions make your code more readable and allow you to handle application-specific errors distinctly from generic Python errors.

Creating Basic Custom Exceptions

To create a custom exception, simply inherit from the Exception class (or any of its subclasses). Here's the simplest form:

# Simple custom exception
class InsufficientFundsError(Exception):
    """Raised when a bank account has insufficient funds"""
    pass

# Using the custom exception
class BankAccount:
    def __init__(self, balance):
        self.balance = balance
    
    def withdraw(self, amount):
        if amount > self.balance:
            raise InsufficientFundsError(f"Cannot withdraw ${amount}. Balance: ${self.balance}")
        self.balance -= amount
        return self.balance

# Usage
account = BankAccount(100)
try:
    account.withdraw(150)
except InsufficientFundsError as e:
    print(f"Transaction failed: {e}")
    # Output: Transaction failed: Cannot withdraw $150. Balance: $100

Custom Exceptions with Additional Attributes

You can make custom exceptions more powerful by adding attributes that provide context about the error:

class ValidationError(Exception):
    """Raised when data validation fails"""
    
    def __init__(self, field, value, message):
        self.field = field
        self.value = value
        self.message = message
        super().__init__(self.message)
    
    def __str__(self):
        return f"Validation failed for '{self.field}' with value '{self.value}': {self.message}"

class AgeValidationError(ValidationError):
    """Specific validation error for age fields"""
    
    def __init__(self, value):
        super().__init__(
            field="age",
            value=value,
            message=f"Age must be between 0 and 150, got {value}"
        )

# Usage in a user registration system
def register_user(name, age, email):
    if not isinstance(age, int) or age < 0 or age > 150:
        raise AgeValidationError(age)
    
    if not email or '@' not in email:
        raise ValidationError("email", email, "Invalid email format")
    
    print(f"User {name} registered successfully!")

# Test cases
try:
    register_user("Alice", 25, "alice@example.com")
except ValidationError as e:
    print(f"Registration failed: {e}")

try:
    register_user("Bob", 200, "bob@example.com")
except AgeValidationError as e:
    print(f"Registration failed: {e}")
    print(f"Field: {e.field}, Value: {e.value}")
    # Output: Registration failed: Validation failed for 'age' with value '200': Age must be between 0 and 150, got 200

Custom Exception Hierarchy

For larger applications, creating an exception hierarchy helps organize error handling:

# Base exception for all application errors
class AppError(Exception):
    """Base exception for all application errors"""
    pass

# Database-related exceptions
class DatabaseError(AppError):
    """Base exception for database errors"""
    pass

class ConnectionError(DatabaseError):
    """Database connection failed"""
    pass

class QueryError(DatabaseError):
    """Database query execution failed"""
    pass

# Authentication-related exceptions
class AuthenticationError(AppError):
    """Base exception for authentication errors"""
    pass

class InvalidCredentialsError(AuthenticationError):
    """User credentials are invalid"""
    pass

class SessionExpiredError(AuthenticationError):
    """User session has expired"""
    pass

# Using the hierarchy
def login_user(username, password):
    if username != "admin":
        raise InvalidCredentialsError(f"Unknown user: {username}")
    if password != "secret":
        raise InvalidCredentialsError("Invalid password")
    return {"user": username, "session": "abc123"}

def fetch_user_data(session_id):
    if session_id != "abc123":
        raise SessionExpiredError("Please log in again")
    return {"name": "Admin", "role": "administrator"}

# Error handling with hierarchy
try:
    user = login_user("admin", "wrong_password")
    data = fetch_user_data(user["session"])
except InvalidCredentialsError as e:
    print(f"Login failed: {e}")
except SessionExpiredError as e:
    print(f"Session error: {e}")
except AuthenticationError as e:
    # Catches any authentication error not caught above
    print(f"Authentication error: {e}")
except AppError as e:
    # Catches any application error not caught above
    print(f"Application error: {e}")

💡 Design Tip: When creating custom exceptions, name them descriptively and follow the convention of ending the name with "Error" (e.g., PaymentError, ConfigurationError). This makes their purpose immediately clear.

Exception Chaining

Exception chaining allows you to preserve the original exception's context when raising a new exception. This is crucial for debugging because it shows the complete chain of what went wrong.

Implicit Exception Chaining

When an exception occurs while handling another exception, Python automatically chains them:

def process_data(data):
    try:
        # Intentionally cause an error
        result = int(data)
    except ValueError:
        # Another error occurs while handling the first
        undefined_variable  # NameError
    
try:
    process_data("invalid")
except Exception as e:
    print(type(e))
    # Output shows both exceptions:
    # During handling of the above exception (ValueError), another exception occurred:
    # NameError: name 'undefined_variable' is not defined

Explicit Exception Chaining with 'from'

The from keyword lets you explicitly chain exceptions, showing that one error directly caused another:

class DataProcessingError(Exception):
    """Custom exception for data processing errors"""
    pass

def parse_user_age(age_string):
    """Parse age from string, raising custom exception on failure"""
    try:
        age = int(age_string)
        if age < 0:
            raise ValueError("Age cannot be negative")
        return age
    except ValueError as e:
        # Chain the original ValueError to our custom exception
        raise DataProcessingError(f"Failed to parse age: {age_string}") from e

# Usage
try:
    user_age = parse_user_age("twenty-five")
except DataProcessingError as e:
    print(f"Error: {e}")
    print(f"Original cause: {e.__cause__}")
    # Output:
    # Error: Failed to parse age: twenty-five
    # Original cause: invalid literal for int() with base 10: 'twenty-five'

Suppressing Exception Context with 'from None'

Sometimes you want to raise a new exception without showing the original exception chain. Use from None for this:

def get_user_config(user_id):
    """Fetch user configuration, raising clean errors"""
    try:
        # Simulate database query
        config = database.query(f"SELECT * FROM configs WHERE user_id = {user_id}")
        return config
    except DatabaseConnectionError:
        # Replace low-level error with user-friendly message
        raise RuntimeError("Configuration service is temporarily unavailable") from None

# Without 'from None', users would see the internal DatabaseConnectionError
# With 'from None', they only see the clean RuntimeError message

Real-World Exception Chaining Example

import json

class ConfigurationError(Exception):
    """Raised when configuration loading fails"""
    pass

class InvalidConfigFormatError(ConfigurationError):
    """Raised when configuration file format is invalid"""
    pass

class MissingConfigValueError(ConfigurationError):
    """Raised when required configuration value is missing"""
    pass

def load_application_config(filepath):
    """
    Load application configuration with proper error chaining
    """
    try:
        with open(filepath, 'r') as file:
            content = file.read()
    except FileNotFoundError as e:
        raise ConfigurationError(
            f"Configuration file not found: {filepath}"
        ) from e
    except PermissionError as e:
        raise ConfigurationError(
            f"Permission denied reading config: {filepath}"
        ) from e
    
    try:
        config = json.loads(content)
    except json.JSONDecodeError as e:
        raise InvalidConfigFormatError(
            f"Configuration file contains invalid JSON: {filepath}"
        ) from e
    
    # Validate required fields
    required_fields = ['database_url', 'api_key', 'port']
    for field in required_fields:
        if field not in config:
            raise MissingConfigValueError(
                f"Required configuration field missing: {field}"
            ) from None  # No chaining needed for validation errors
    
    return config

# Usage with comprehensive error handling
def start_application():
    try:
        config = load_application_config("config.json")
        print(f"Application started on port {config['port']}")
    except InvalidConfigFormatError as e:
        print(f"Configuration format error: {e}")
        print(f"Original JSON error: {e.__cause__}")
    except MissingConfigValueError as e:
        print(f"Configuration incomplete: {e}")
    except ConfigurationError as e:
        print(f"Configuration error: {e}")
        if e.__cause__:
            print(f"Underlying cause: {e.__cause__}")
    except Exception as e:
        print(f"Unexpected error: {e}")

start_application()

Accessing Exception Chain Information

Python provides special attributes to inspect the exception chain:

try:
    try:
        1 / 0
    except ZeroDivisionError as e:
        raise ValueError("Invalid calculation") from e
except ValueError as e:
    print(f"Current exception: {e}")
    print(f"Direct cause (__cause__): {e.__cause__}")
    print(f"Exception context (__context__): {e.__context__}")
    print(f"Suppress context (__suppress_context__): {e.__suppress_context__}")
    
    # Output:
    # Current exception: Invalid calculation
    # Direct cause (__cause__): division by zero
    # Exception context (__context__): division by zero
    # Suppress context (__suppress_context__): False

Advanced Error Handling Patterns

Retrying Operations with Exponential Backoff

import time
import random

class NetworkError(Exception):
    """Simulated network error"""
    pass

def fetch_data_from_api(url):
    """Simulate API call that might fail"""
    if random.random() < 0.7:  # 70% chance of failure
        raise NetworkError("Connection timeout")
    return {"data": "Success!"}

def retry_with_backoff(func, max_retries=3, initial_delay=1):
    """
    Retry a function with exponential backoff
    """
    delay = initial_delay
    
    for attempt in range(max_retries):
        try:
            result = func()
            print(f"Success on attempt {attempt + 1}")
            return result
        except NetworkError as e:
            if attempt == max_retries - 1:
                # Last attempt failed, re-raise the exception
                print(f"All {max_retries} attempts failed")
                raise
            
            print(f"Attempt {attempt + 1} failed: {e}")
            print(f"Retrying in {delay} seconds...")
            time.sleep(delay)
            delay *= 2  # Exponential backoff

# Usage
try:
    data = retry_with_backoff(
        lambda: fetch_data_from_api("https://api.example.com"),
        max_retries=5
    )
    print(f"Retrieved data: {data}")
except NetworkError as e:
    print(f"Failed to fetch data after multiple retries: {e}")

Context Managers for Resource Management

class DatabaseConnection:
    """Custom context manager with exception handling"""
    
    def __init__(self, connection_string):
        self.connection_string = connection_string
        self.connection = None
    
    def __enter__(self):
        print(f"Opening connection to {self.connection_string}")
        self.connection = self._connect()
        return self.connection
    
    def __exit__(self, exc_type, exc_value, traceback):
        """
        Called when exiting the 'with' block
        exc_type: Type of exception (if any)
        exc_value: Exception instance (if any)
        traceback: Traceback object (if any)
        """
        if exc_type is not None:
            print(f"Exception occurred: {exc_type.__name__}: {exc_value}")
            # Perform rollback on error
            if self.connection:
                print("Rolling back transaction")
                self.connection.rollback()
        else:
            # Commit on success
            if self.connection:
                print("Committing transaction")
                self.connection.commit()
        
        # Always close connection
        if self.connection:
            print("Closing connection")
            self.connection.close()
        
        # Return False to propagate exception, True to suppress it
        return False
    
    def _connect(self):
        # Simulate connection
        return {"status": "connected"}

# Usage
try:
    with DatabaseConnection("postgresql://localhost/mydb") as conn:
        print("Executing database operations...")
        # Simulate an error
        raise ValueError("Something went wrong!")
except ValueError as e:
    print(f"Caught exception: {e}")

# Output shows proper cleanup even with exception:
# Opening connection to postgresql://localhost/mydb
# Executing database operations...
# Exception occurred: ValueError: Something went wrong!
# Rolling back transaction
# Closing connection
# Caught exception: Something went wrong!

Logging Exceptions Properly

import logging
import traceback

# Configure logging
logging.basicConfig(
    level=logging.INFO,
    format='%(asctime)s - %(name)s - %(levelname)s - %(message)s'
)
logger = logging.getLogger(__name__)

def process_payment(amount, card_number):
    """Process payment with comprehensive logging"""
    try:
        logger.info(f"Processing payment of ${amount}")
        
        # Validate input
        if amount <= 0:
            raise ValueError(f"Invalid amount: {amount}")
        
        if len(card_number) != 16:
            raise ValueError(f"Invalid card number length: {len(card_number)}")
        
        # Simulate payment processing
        if card_number.startswith("0000"):
            raise RuntimeError("Payment gateway error: Card declined")
        
        logger.info(f"Payment of ${amount} processed successfully")
        return {"status": "success", "transaction_id": "TXN123456"}
        
    except ValueError as e:
        # Log validation errors at WARNING level
        logger.warning(f"Validation error: {e}")
        raise
    except RuntimeError as e:
        # Log payment gateway errors at ERROR level with full traceback
        logger.error(f"Payment processing failed: {e}")
        logger.error(traceback.format_exc())
        raise
    except Exception as e:
        # Log unexpected errors at CRITICAL level
        logger.critical(f"Unexpected error in payment processing: {e}")
        logger.critical(traceback.format_exc())
        raise

# Usage
try:
    result = process_payment(100.00, "0000111122223333")
except Exception as e:
    print(f"Payment failed: {e}")

Edge Cases and Common Pitfalls

Edge Case 1: Empty Except Blocks

❌ Anti-Pattern: Silent Failures

# BAD: Silently swallowing exceptions
try:
    risky_operation()
except:
    pass  # This hides all errors, including system exits!

# GOOD: At minimum, log the error
try:
    risky_operation()
except Exception as e:
    logger.error(f"Operation failed: {e}")
    # Decide whether to re-raise or handle

Edge Case 2: Catching Too Broad Exceptions

# BAD: Catching too broadly
try:
    user_age = int(input("Enter age: "))
    result = 100 / user_age
except Exception as e:
    print("An error occurred")
    # What error? ValueError? ZeroDivisionError? Something else?

# GOOD: Catch specific exceptions
try:
    user_age = int(input("Enter age: "))
    result = 100 / user_age
except ValueError:
    print("Please enter a valid number")
except ZeroDivisionError:
    print("Age cannot be zero")
except KeyboardInterrupt:
    print("\nOperation cancelled by user")
    raise  # Re-raise to allow proper cleanup

Edge Case 3: Exceptions in Finally Block

def problematic_finally():
    try:
        print("Doing work...")
        raise ValueError("Original error")
    finally:
        # If exception occurs here, it masks the original error!
        raise RuntimeError("Finally block error")

try:
    problematic_finally()
except RuntimeError as e:
    print(f"Caught: {e}")
    # The original ValueError is lost!

# BETTER: Protect finally blocks
def safe_finally():
    try:
        print("Doing work...")
        raise ValueError("Original error")
    finally:
        try:
            # Wrap cleanup code that might fail
            cleanup_operation()
        except Exception as e:
            logger.error(f"Cleanup failed: {e}")
            # Don't re-raise, preserving original exception

Edge Case 4: Return in Try-Except-Finally

def confusing_returns():
    try:
        return "from try"
    except:
        return "from except"
    finally:
        return "from finally"  # This ALWAYS wins!

result = confusing_returns()
print(result)  # Output: "from finally"

# BETTER: Avoid returns in finally
def clear_returns():
    result = None
    try:
        result = "from try"
    except:
        result = "from except"
    finally:
        # Use finally only for cleanup, not return values
        cleanup()
    return result

Edge Case 5: Mutable Default Arguments in Exception Handlers

# CAREFUL: Exception instances are reused
def handle_errors(errors=[]):  # Mutable default!
    try:
        risky_operation()
    except ValueError as e:
        errors.append(e)
    return errors

# First call
errors1 = handle_errors()
print(len(errors1))  # 1

# Second call (without arguments)
errors2 = handle_errors()
print(len(errors2))  # 2 (shares the same list!)

# CORRECT: Use None as default
def handle_errors_correct(errors=None):
    if errors is None:
        errors = []
    try:
        risky_operation()
    except ValueError as e:
        errors.append(e)
    return errors

Real-World Application: Building a Robust File Processor

Let's put everything together in a realistic example that processes multiple files with comprehensive error handling:

import os
import json
import logging
from typing import List, Dict, Any
from datetime import datetime

# Configure logging
logging.basicConfig(
    level=logging.INFO,
    format='%(asctime)s - %(levelname)s - %(message)s',
    handlers=[
        logging.FileHandler('file_processor.log'),
        logging.StreamHandler()
    ]
)
logger = logging.getLogger(__name__)

# Custom exceptions
class FileProcessingError(Exception):
    """Base exception for file processing errors"""
    pass

class InvalidFileFormatError(FileProcessingError):
    """File format is not supported"""
    pass

class DataValidationError(FileProcessingError):
    """Data validation failed"""
    pass

class ProcessingResult:
    """Container for processing results"""
    def __init__(self):
        self.successful: List[str] = []
        self.failed: Dict[str, str] = {}
        self.skipped: List[str] = []
    
    def summary(self) -> str:
        return (
            f"Processed: {len(self.successful)} successful, "
            f"{len(self.failed)} failed, {len(self.skipped)} skipped"
        )

class FileProcessor:
    """Robust file processor with comprehensive error handling"""
    
    SUPPORTED_FORMATS = ['.json', '.txt', '.csv']
    MAX_FILE_SIZE = 10 * 1024 * 1024  # 10 MB
    
    def __init__(self, input_dir: str, output_dir: str):
        self.input_dir = input_dir
        self.output_dir = output_dir
        self._validate_directories()
    
    def _validate_directories(self):
        """Validate input and output directories"""
        try:
            if not os.path.exists(self.input_dir):
                raise FileNotFoundError(f"Input directory not found: {self.input_dir}")
            
            if not os.path.exists(self.output_dir):
                logger.info(f"Creating output directory: {self.output_dir}")
                os.makedirs(self.output_dir)
        except PermissionError as e:
            raise FileProcessingError(
                f"Permission denied accessing directories"
            ) from e
    
    def process_files(self, filenames: List[str]) -> ProcessingResult:
        """
        Process multiple files with individual error handling
        """
        result = ProcessingResult()
        
        logger.info(f"Starting batch processing of {len(filenames)} files")
        
        for filename in filenames:
            try:
                self._process_single_file(filename)
                result.successful.append(filename)
                logger.info(f"✓ Successfully processed: {filename}")
                
            except InvalidFileFormatError as e:
                result.skipped.append(filename)
                logger.warning(f"⊘ Skipped {filename}: {e}")
                
            except DataValidationError as e:
                result.failed[filename] = str(e)
                logger.error(f"✗ Validation failed for {filename}: {e}")
                
            except FileProcessingError as e:
                result.failed[filename] = str(e)
                logger.error(f"✗ Processing failed for {filename}: {e}")
                if e.__cause__:
                    logger.error(f"  Caused by: {e.__cause__}")
                    
            except Exception as e:
                # Catch unexpected errors
                result.failed[filename] = f"Unexpected error: {str(e)}"
                logger.critical(f"✗ Unexpected error processing {filename}: {e}")
                logger.exception("Full traceback:")
        
        logger.info(f"Batch processing complete: {result.summary()}")
        return result
    
    def _process_single_file(self, filename: str):
        """
        Process a single file with detailed error handling
        """
        filepath = os.path.join(self.input_dir, filename)
        
        # Validate file format
        _, ext = os.path.splitext(filename)
        if ext.lower() not in self.SUPPORTED_FORMATS:
            raise InvalidFileFormatError(
                f"Unsupported format: {ext}. Supported: {self.SUPPORTED_FORMATS}"
            )
        
        # Check file size
        try:
            file_size = os.path.getsize(filepath)
            if file_size > self.MAX_FILE_SIZE:
                raise FileProcessingError(
                    f"File too large: {file_size} bytes (max: {self.MAX_FILE_SIZE})"
                )
        except OSError as e:
            raise FileProcessingError(f"Cannot access file: {filename}") from e
        
        # Read and process file
        content = None
        try:
            with open(filepath, 'r', encoding='utf-8') as f:
                content = f.read()
        except UnicodeDecodeError as e:
            raise FileProcessingError(
                f"File encoding error: {filename} is not UTF-8"
            ) from e
        except PermissionError as e:
            raise FileProcessingError(
                f"Permission denied reading: {filename}"
            ) from e
        except Exception as e:
            raise FileProcessingError(
                f"Error reading file: {filename}"
            ) from e
        
        # Parse and validate based on format
        try:
            if ext == '.json':
                data = self._process_json(content, filename)
            elif ext == '.csv':
                data = self._process_csv(content, filename)
            else:
                data = self._process_text(content, filename)
        except Exception as e:
            raise DataValidationError(
                f"Failed to parse {ext} format"
            ) from e
        
        # Write output
        self._write_output(filename, data)
    
    def _process_json(self, content: str, filename: str) -> Dict:
        """Process JSON file with validation"""
        try:
            data = json.loads(content)
        except json.JSONDecodeError as e:
            raise DataValidationError(
                f"Invalid JSON in {filename} at line {e.lineno}"
            ) from e
        
        # Validate required fields
        if not isinstance(data, dict):
            raise DataValidationError("JSON must be an object")
        
        if 'id' not in data:
            raise DataValidationError("Missing required field: 'id'")
        
        return data
    
    def _process_csv(self, content: str, filename: str) -> List[Dict]:
        """Process CSV file"""
        lines = content.strip().split('\n')
        if len(lines) < 2:
            raise DataValidationError("CSV must have header and at least one data row")
        
        headers = lines[0].split(',')
        data = []
        
        for i, line in enumerate(lines[1:], start=2):
            values = line.split(',')
            if len(values) != len(headers):
                raise DataValidationError(
                    f"Row {i} has {len(values)} values, expected {len(headers)}"
                )
            data.append(dict(zip(headers, values)))
        
        return data
    
    def _process_text(self, content: str, filename: str) -> Dict:
        """Process text file"""
        return {
            'filename': filename,
            'lines': len(content.split('\n')),
            'characters': len(content),
            'content': content
        }
    
    def _write_output(self, filename: str, data: Any):
        """Write processed data to output file"""
        output_filename = f"processed_{filename}"
        output_path = os.path.join(self.output_dir, output_filename)
        
        try:
            with open(output_path, 'w', encoding='utf-8') as f:
                json.dump({
                    'source': filename,
                    'processed_at': datetime.now().isoformat(),
                    'data': data
                }, f, indent=2)
        except Exception as e:
            raise FileProcessingError(
                f"Failed to write output file"
            ) from e

# Usage example
def main():
    processor = FileProcessor(
        input_dir='./input_files',
        output_dir='./output_files'
    )
    
    files_to_process = [
        'data.json',
        'users.csv',
        'notes.txt',
        'invalid.xml',  # Will be skipped
        'corrupted.json'  # Will fail
    ]
    
    try:
        result = processor.process_files(files_to_process)
        
        print("\n" + "="*50)
        print("PROCESSING SUMMARY")
        print("="*50)
        print(result.summary())
        
        if result.successful:
            print(f"\n✓ Successful: {', '.join(result.successful)}")
        
        if result.skipped:
            print(f"\n⊘ Skipped: {', '.join(result.skipped)}")
        
        if result.failed:
            print("\n✗ Failed:")
            for filename, error in result.failed.items():
                print(f"  - {filename}: {error}")
                
    except FileProcessingError as e:
        logger.critical(f"Fatal error: {e}")
        if e.__cause__:
            logger.critical(f"Caused by: {e.__cause__}")
    except Exception as e:
        logger.critical(f"Unexpected fatal error: {e}")
        logger.exception("Full traceback:")

if __name__ == "__main__":
    main()

Best Practices Summary

✅ DO:

• Catch specific exceptions rather than generic Exception
• Use custom exceptions for application-specific errors
• Chain exceptions to preserve error context (raise ... from e)
• Always clean up resources in finally blocks or use context managers
• Log exceptions with appropriate severity levels
• Provide helpful error messages that guide users toward solutions
• Re-raise exceptions when you can't handle them properly
• Document what exceptions your functions can raise

❌ DON'T:

• Use bare except: clauses that catch everything
• Silently ignore exceptions with empty except blocks
• Catch BaseException (catches system exits and keyboard interrupts)
• Return values from finally blocks
• Raise exceptions in finally blocks (masks original errors)
• Use exceptions for normal control flow
• Create exception hierarchies that are too deep or complex
• Assume all errors are programming mistakes (some are expected conditions)

Performance Considerations

Exception handling has performance implications. Understanding them helps you write efficient code:

import time

# Scenario 1: LBYL (Look Before You Leap) - Check first
def lbyl_approach(data, key):
    if key in data:
        return data[key]
    return None

# Scenario 2: EAFP (Easier to Ask Forgiveness than Permission) - Try and catch
def eafp_approach(data, key):
    try:
        return data[key]
    except KeyError:
        return None

# Performance comparison
data = {str(i): i for i in range(10000)}
existing_key = "5000"
missing_key = "99999"

# Test with existing key (successful path)
start = time.time()
for _ in range(100000):
    lbyl_approach(data, existing_key)
print(f"LBYL (key exists): {time.time() - start:.4f}s")

start = time.time()
for _ in range(100000):
    eafp_approach(data, existing_key)
print(f"EAFP (key exists): {time.time() - start:.4f}s")

# Test with missing key (exception path)
start = time.time()
for _ in range(100000):
    lbyl_approach(data, missing_key)
print(f"LBYL (key missing): {time.time() - start:.4f}s")

start = time.time()
for _ in range(100000):
    eafp_approach(data, missing_key)
print(f"EAFP (key missing): {time.time() - start:.4f}s")

# Generally:
# - EAFP is faster when exceptions are rare (success is common)
# - LBYL is faster when errors are frequent
# - EAFP is more Pythonic and handles race conditions better

💡 Python Philosophy: Python favors EAFP (Easier to Ask Forgiveness than Permission) over LBYL (Look Before You Leap). The rationale is that in concurrent environments, checking first doesn't guarantee the state won't change before you act. EAFP code is also often cleaner and more readable.

Conclusion

Error handling is not about eliminating errors—it's about managing them gracefully. Well-designed exception handling makes your code resilient, maintainable, and user-friendly. By understanding Python's exception hierarchy, using try-except-else-finally blocks appropriately, creating meaningful custom exceptions, and leveraging exception chaining, you transform fragile scripts into robust applications.

Remember that exceptions are not failures—they're Python's way of communicating that something unexpected happened. Your job as a developer is to anticipate these situations, handle them appropriately, and guide your program toward recovery or graceful degradation. The patterns and practices covered in this guide provide the foundation for building production-quality Python applications that handle the real world's unpredictability with confidence.