Publications / Books

Linux Distributed Systems Programming with Python

A Lab-Driven Systems-Level Guide to Building Reliable Distributed Systems

By Weqaar Janjua · First edition · 2026

Buy the Book

Website checkout coming soon.

Book cover

100%
Front cover of Linux Distributed Systems Programming with Python

Scroll or pinch to zoom. Drag to pan. Double-click to zoom or reset. Escape closes the viewer.

About the Book

Build a Python web service with REST, GraphQL and WebSocket APIs, then explore the mechanisms behind distributed computing. Follow Python execution through bytecode, memory and Linux system calls. Program sockets, investigate the network stack and connect application behavior to kernel activity with eBPF.

Follow a developing service through 39 chapters and their associated labs. Work with messaging, concurrency, Raft consensus, replicated storage and resource-aware scheduling, including CPU, GPU and NUMA placement for cloud and AI/HPC environments. Apply production-engineering practices with Microsoft Azure, OpenStack, Kubernetes and OpenTelemetry. Required lab checks run locally without a cloud subscription; live cloud exercises use separately provisioned infrastructure.

Topics

Table of Contents

First edition · 39 chapters in 11 parts, plus appendices. Expand a chapter to see its sections and subsections.

Contents headings only; the full text is not available on this website. Page numbers are omitted.

Preface
  • About this book
  • Terms used in this book
  • Intended readership
  • Book organization
  • Ethics and AI-assisted development
  • Acknowledgments
  1. Part I Production engineering

    1. Chapter 1 The engineering process
      1. 1.1 Command-line and version-control prerequisites
      2. 1.2 Environment drift
      3. 1.3 Continuous integration and quality gates
        1. 1.3.1 Builds and artifacts
        2. 1.3.2 Pipelines, jobs and steps
        3. 1.3.3 Workers, agents and nodes
        4. 1.3.4 Formatting, linting, types and tests
        5. 1.3.5 Stages of a CI run
        6. 1.3.6 Hidden failures in passing builds
      4. 1.4 The development environment
        1. 1.4.1 Installing Visual Studio Code
        2. 1.4.2 Installing the extensions
      5. 1.5 Software testing
        1. 1.5.1 Unit tests
        2. 1.5.2 Functional tests
        3. 1.5.3 Confirming a fix with a failing test
      6. 1.6 The coding standard
        1. 1.6.1 Classes, functions and interfaces
        2. 1.6.2 Type hints and the Any type
        3. 1.6.3 Error handling and resource cleanup
      7. 1.7 AI coding agents
      8. 1.8 Local and privacy-preserving agents
        1. 1.8.1 A vendor-neutral local setup
        2. 1.8.2 Local inference hardware
        3. 1.8.3 Prompt data flow
        4. 1.8.4 Verifying the LLM file and its license
        5. 1.8.5 Context leakage and resource limits
        6. 1.8.6 Accepting changes from a local agent
      9. 1.9 Reading gate results
      10. 1.10 Building reproducible environments
      11. 1.11 Installing the toolchain
      12. 1.12 Configuring the gates in pyproject.toml
      13. 1.13 Distinguishing code violations from tool crashes
      14. 1.14 Code coverage as an acceptance gate
      15. 1.15 Automating gate execution in continuous integration
      16. 1.16 Archiving quality reports and build artifacts
      17. 1.17 Jobs and job status
    2. Chapter 2 Packaging and building a Python project
      1. 2.1 Project templates
      2. 2.2 Source distributions and wheels
      3. 2.3 Wheel contents and compatibility tags
      4. 2.4 Binary compatibility and manylinux wheels
      5. 2.5 Standalone executables
      6. 2.6 Bundling with PyInstaller and compiling with Nuitka
      7. 2.7 Building on each target operating system
      8. 2.8 Defining a build matrix
      9. 2.9 Reading an ELF executable
      10. 2.10 Windows Portable Executable files
      11. 2.11 Project metadata and build configuration
      12. 2.12 The src layout and testing the installed package
      13. 2.13 Version ranges and pinned versions
      14. 2.14 Reproducible builds, SBOMs and signatures
      15. 2.15 Publishing to a private package index
      16. 2.16 Building documentation with Sphinx
        1. 2.16.1 Writing docstrings and comments
        2. 2.16.2 Previewing the documentation site
        3. 2.16.3 Comparing Sphinx with MkDocs
    3. Chapter 3 Code quality and compliance
      1. 3.1 Architecture of a quality gate suite
      2. 3.2 Interpreting tool exit codes with adapters
      3. 3.3 Unified gate execution across local workstations and CI
      4. 3.4 Reporting results in JUnit XML
      5. 3.5 Enforcing coverage thresholds with pytest-cov
      6. 3.6 Writing tests with pytest and unittest
      7. 3.7 Mocks, fakes and test doubles
      8. 3.8 Auditing dependencies for vulnerabilities and licenses
      9. 3.9 Suppressing false positives
      10. 3.10 Publishing quality reports in Azure Pipelines
      11. 3.11 Formatting, docstrings and type annotations
      12. 3.12 Testing the documentation build
    4. Chapter 4 Command line tools
      1. 4.1 Separating offline, service and cloud commands
      2. 4.2 Standard streams and interactive terminals
      3. 4.3 Output stream buffering in standard streams
        1. 4.3.1 Pipe capacity and backpressure
      4. 4.4 Defining subcommands with argparse
      5. 4.5 Building commands with Typer
      6. 4.6 Configuration source precedence
      7. 4.7 CLI command exit codes and error status reporting
        1. 4.7.1 Handling Ctrl-C and termination signals
      8. 4.8 Human-readable and machine-readable output
      9. 4.9 Authenticating with bearer tokens
      10. 4.10 Calling the job API over HTTP
      11. 4.11 Retries, backoff and command deadlines
      12. 4.12 Shell completion and help
      13. 4.13 Testing commands without a network
    5. Chapter 5 Object-oriented design in Python
      1. 5.1 The Python object model and memory attributes
        1. 5.1.1 Methods, functions and self
        2. 5.1.2 Special methods and Python's built-in operations
        3. 5.1.3 Inspecting objects in the interpreter
      2. 5.2 Names, identity and aliasing
      3. 5.3 Defining domain values with dataclasses
      4. 5.4 Encapsulation and private attributes
        1. 5.4.1 Properties and descriptors
      5. 5.5 Interfaces with protocols and abstract base classes
      6. 5.6 Composition and dependency injection
      7. 5.7 Inheritance, polymorphism and method resolution order
      8. 5.8 Choosing between lists and dictionaries
      9. 5.9 Function and method decorators
      10. 5.10 Reflection and metaclasses
      11. 5.11 Organizing relay into layers of objects
    6. Chapter 6 Trees and graphs for distributed systems
      1. 6.1 Measuring operation cost with Big-O notation
      2. 6.2 Trees and ordered indexes
        1. 6.2.1 Rooted trees
        2. 6.2.2 Tree traversal
        3. 6.2.3 Updating and removing nodes
        4. 6.2.4 Common tree structures
        5. 6.2.5 Deleting from a binary search tree
        6. 6.2.6 Rebuilding and self-balancing trees
        7. 6.2.7 Balancing a red-black tree
      3. 6.3 Representing graphs in Python
        1. 6.3.1 Graph terms and changing a graph
        2. 6.3.2 The graphlib module and graph interfaces
      4. 6.4 Depth-first and breadth-first graph search
      5. 6.5 Shortest paths on weighted graphs
        1. 6.5.1 Dijkstra's algorithm and edge relaxation
        2. 6.5.2 Bellman-Ford and negative cycles
      6. 6.6 OSPF route calculation
      7. 6.7 Dependency graphs and topological sorting
      8. 6.8 Trees and graphs in distributed systems
      9. 6.9 Testing correctness before optimizing
    7. Chapter 7 Debugging, profiling and benchmarking
      1. 7.1 Defining the problem
        1. 7.1.1 Overhead, permissions and sensitive data
      2. 7.2 Debugging Python code with pdb
        1. 7.2.1 Reproducing the failure first
        2. 7.2.2 Starting the debugger from code and after a crash
        3. 7.2.3 Inspecting live objects
        4. 7.2.4 The dir function and the __debug__ constant
      3. 7.3 Timing one operation
        1. 7.3.1 Choosing a clock
        2. 7.3.2 Timing with timeit
      4. 7.4 Profiling Python code
        1. 7.4.1 Profiling every call with cProfile
        2. 7.4.2 Choosing what to profile and the cost of profiling
        3. 7.4.3 Tracing memory allocation with tracemalloc
      5. 7.5 Benchmarking with pyperf
        1. 7.5.1 Defining what the benchmark measures
        2. 7.5.2 Writing benchmarks with the pyperf Runner
        3. 7.5.3 Repeated measurements in pyperf
        4. 7.5.4 Running and saving benchmarks
        5. 7.5.5 Preparing the benchmark machine
        6. 7.5.6 Reading the spread of benchmark results
        7. 7.5.7 Comparing a baseline with a candidate
        8. 7.5.8 Common pyperf mistakes
      6. 7.6 Debugging native code with GDB
        1. 7.6.1 Building with debug symbols
        2. 7.6.2 Attaching to a process and reading core dumps
      7. 7.7 Tracing system calls with strace
      8. 7.8 Inspecting open files, sockets and process state
      9. 7.9 Counting events and sampling with perf
      10. 7.10 Tracing kernel events with ftrace
      11. 7.11 Investigating a worker blocked on I/O
  2. Part II The Azure platform

    1. Chapter 8 Azure infrastructure for distributed systems
      1. 8.1 IaaS, PaaS and SaaS cloud service models
        1. 8.1.1 Service model selection for SigRaft
      2. 8.2 Comparing Azure cloud services to on-premises counterparts
      3. 8.3 Azure resource scopes and inherited policy
      4. 8.4 Managing resources with Azure Resource Manager
      5. 8.5 Identities and role-based access control
      6. 8.6 Control-plane and data-plane permissions
      7. 8.7 Regions and availability zones
      8. 8.8 Virtual networks, private endpoints and DNS
      9. 8.9 Controlling cost and cleaning up resources
      10. 8.10 Diagnosing a failed deployment
      11. 8.11 Declarative infrastructure with Bicep
        1. 8.11.1 Why we use Bicep
        2. 8.11.2 How we use Bicep
        3. 8.11.3 Bicep in the SigRaft project and labs
      12. 8.12 Describing SigRaft infrastructure in Bicep
        1. 8.12.1 Parameters and parameter files
        2. 8.12.2 Modules, dependencies and deployment scopes
      13. 8.13 Containers, Kubernetes and workload identity
        1. 8.13.1 Containers and operating-system virtualization
        2. 8.13.2 Types of container technologies
        3. 8.13.3 Kubernetes and container orchestration
        4. 8.13.4 Container registries and workload identity in Azure
        5. 8.13.5 Container and Kubernetes coverage across the labs
      14. 8.14 Deploying and tearing down infrastructure
      15. 8.15 Simulating cloud deployments with local resource graphs
    2. Chapter 9 The Azure Python SDKs
      1. 9.1 Management and data-plane packages
      2. 9.2 Portable client adapter design across cloud and on-prem backends
      3. 9.3 Choosing credentials for each deployment
      4. 9.4 Creating and closing clients
      5. 9.5 Reading paged results
      6. 9.6 Long-running operations
      7. 9.7 Retries and idempotent operations
      8. 9.8 Handling request and response failures
      9. 9.9 Asynchronous clients
      10. 9.10 Logging SDK calls without leaking secrets
      11. 9.11 Testing adapters with protocols and fakes
  3. Part III Python and the operating system

    1. Chapter 10 Python program execution
      1. 10.1 From source text to a running process
      2. 10.2 The Python language, implementations and packages
      3. 10.3 Starting the Python interpreter on Linux
      4. 10.4 From source text to a syntax tree
        1. 10.4.1 Tokenizing source code
        2. 10.4.2 Parsing tokens into a syntax tree
        3. 10.4.3 Traversing and transforming the syntax tree
      5. 10.5 Processor instructions and virtual-machine instructions
        1. 10.5.1 Encoding integer addition in RISC-V
        2. 10.5.2 Implementing ADD with digital logic
        3. 10.5.3 Instructions, opcodes and operands
        4. 10.5.4 Stacks and stack machines
      6. 10.6 Bytecode execution in a virtual machine
      7. 10.7 Reading CPython bytecode
        1. 10.7.1 Adaptive specialization and JIT compilation
      8. 10.8 Frames, objects and function calls
      9. 10.9 Portability across CPU architectures
      10. 10.10 Python on x86-64 Linux
      11. 10.11 C extensions and Cython
      12. 10.12 Inspecting source, bytecode and native modules
    2. Chapter 11 Processes, concurrency and NUMA performance
      1. 11.1 Linux thread scheduling and container CPU limits
      2. 11.2 Concurrency versus parallelism
        1. 11.2.1 Multiprocessing versus asynchronous execution
        2. 11.2.2 Native threads versus green threads
        3. 11.2.3 Concurrency and parallelism across the labs
      3. 11.3 Processes, threads and the GIL
      4. 11.4 Worker process communication
        1. 11.4.1 Process start methods and lifetimes
        2. 11.4.2 Queue, SimpleQueue and JoinableQueue
        3. 11.4.3 Managers and shared proxies
        4. 11.4.4 Process pools, result ordering and submission limits
      5. 11.5 Sharing large data through shared memory
      6. 11.6 Multicore performance and memory locality
      7. 11.7 NUMA locality on Linux
        1. 11.7.1 NUMA page placement
        2. 11.7.2 CPU affinity and memory policy
        3. 11.7.3 Python libnuma bindings
        4. 11.7.4 Balancing process workers across nodes
        5. 11.7.5 Measuring locality and cache behavior
      8. 11.8 GPU worker placement
        1. 11.8.1 GPU runtimes and drivers
        2. 11.8.2 PCIe topology and NUMA
        3. 11.8.3 Host-pinned and device memory
        4. 11.8.4 DMA transfers and IOMMU device isolation
      9. 11.9 Remote direct memory access for accelerators
        1. 11.9.1 GPUDirect RDMA requirements
        2. 11.9.2 InfiniBand and RoCE network transports
        3. 11.9.3 RoCEv2 congestion control
        4. 11.9.4 Discovering accelerator devices from sysfs
        5. 11.9.5 Requesting accelerators in Kubernetes
      10. 11.10 Sizing worker pools for blocking calls
      11. 11.11 Asyncio and cooperative scheduling
      12. 11.12 Choosing threads, processes or asyncio
      13. 11.13 Signals and graceful shutdown
    3. Chapter 12 Sockets and the network stack
      1. 12.1 TCP streams and UDP datagrams in Python
        1. 12.1.1 Python sockets and Linux system calls
        2. 12.1.2 Packet loss in TCP and UDP
      2. 12.2 From application message to Ethernet frame
        1. 12.2.1 Receiving and decapsulating a frame
      3. 12.3 Linux packet buffers and sk_buff
        1. 12.3.1 Linked lists and packet queues
        2. 12.3.2 Sending through a net_device
        3. 12.3.3 Receiving from a device
      4. 12.4 Building packets offline with Scapy
      5. 12.5 Name resolution and DNS
      6. 12.6 Creating, binding and connecting sockets
      7. 12.7 The TCP handshake and accept queues on Linux
      8. 12.8 TCP as a byte stream
      9. 12.9 Partial socket reads and writes
      10. 12.10 Waiting for many sockets with epoll
      11. 12.11 TCP retransmission, flow control and congestion control
      12. 12.12 Closing TCP connections, resets and TIME_WAIT
      13. 12.13 Diagnosing DNS, routing and connection problems
      14. 12.14 Testing sockets on loopback
    4. Chapter 13 Packet-level network programming
      1. 13.1 Network layers and packet inspection tools
      2. 13.2 Capture location and offloads
      3. 13.3 Packet headers
      4. 13.4 Checksums and offload
      5. 13.5 MTU, segmentation and fragmentation
      6. 13.6 TCP probes and captures with Scapy
      7. 13.7 Event filtering and counting with eBPF
      8. 13.8 Privileges and isolation
      9. 13.9 Packet loss and connection diagnostics
      10. 13.10 Network fault injection with netem
    5. Chapter 14 Cython and systems-level integration
      1. 14.1 Profiling before choosing native code
      2. 14.2 Compiling a Cython extension
      3. 14.3 The CPython C API and binary compatibility
      4. 14.4 Releasing the GIL for native work
      5. 14.5 C types and memoryviews in a checksum loop
      6. 14.6 Calling Linux C functions and handling errors
      7. 14.7 Memory ownership and error paths
      8. 14.8 Debugging native crashes
      9. 14.9 Platform-specific extension wheels
      10. 14.10 Choosing Cython, ctypes, cffi or Rust bindings
    6. Chapter 15 Engineering CPython bytecode
      1. 15.1 Parsing Python source
        1. 15.1.1 Writing a query language with PyParsing
      2. 15.2 Name resolution and bytecode generation in CPython
      3. 15.3 CPython stack operations and instruction encoding
      4. 15.4 Instruction execution in a CPython frame
      5. 15.5 Connecting a Python function to an opcode
      6. 15.6 Patching a pinned CPython release
      7. 15.7 Testing an interpreter change
      8. 15.8 Processor architecture and distributing bytecode
  4. Part IV Talking between processes

    1. Chapter 16 Protocols and framing
      1. 16.1 Message boundaries in a TCP stream
      2. 16.2 Choosing a framing method
      3. 16.3 Delimiter-based framing and escaping
      4. 16.4 Fixed-width framing
      5. 16.5 Length-prefixed framing
      6. 16.6 Parsing frames incrementally
        1. 16.6.1 Fixed-size ring buffers
      7. 16.7 Handling end of stream
      8. 16.8 Rejecting bad frames and setting read timeouts
      9. 16.9 Framing with asyncio streams
      10. 16.10 Versioning a protocol
      11. 16.11 Testing a frame parser
      12. 16.12 Diagnosing framing problems
    2. Chapter 17 Serialization and wire formats
      1. 17.1 Wire format requirements
      2. 17.2 Encoding Python values as JSON
      3. 17.3 Encoding with MessagePack
      4. 17.4 Protocol Buffers schemas and field numbers
      5. 17.5 Protocol Buffers field encoding
      6. 17.6 Field presence and default values in Protocol Buffers
      7. 17.7 Schema evolution
      8. 17.8 Reviewing schemas and testing compatibility
      9. 17.9 Choosing between JSON and Protocol Buffers
      10. 17.10 Decoding untrusted messages safely
    3. Chapter 18 Data validation
      1. 18.1 Validation boundaries in service architectures
      2. 18.2 Validating input with Pydantic
      3. 18.3 Defining a job submission model
      4. 18.4 Strict mode and type coercion
      5. 18.5 Reporting validation errors without leaking data
      6. 18.6 Handling unknown fields
      7. 18.7 Choosing a command model with a discriminator field
      8. 18.8 Changing validation models over time
      9. 18.9 Validating timestamps, money, identifiers and URLs
      10. 18.10 Limiting the size of untrusted input
      11. 18.11 Parsing throughput and validation overhead
      12. 18.12 Validating configuration at startup
    4. Chapter 19 Remote procedure calls
      1. 19.1 Local and remote calls
      2. 19.2 Designing HTTP resources and status codes
      3. 19.3 Building an HTTP service with FastAPI
      4. 19.4 Running the service with Uvicorn and uvloop
      5. 19.5 gRPC over HTTP/2
      6. 19.6 Timeouts and shared deadlines
      7. 19.7 Transport errors and application errors
      8. 19.8 Retry decisions for failed calls
      9. 19.9 Idempotency keys and stored results
      10. 19.10 Streaming and backpressure
      11. 19.11 Versioning, authentication and logging
      12. 19.12 Testing RPC failures
    5. Chapter 20 Message patterns
      1. 20.1 Message framing and connections in ZeroMQ
      2. 20.2 Contexts, I/O threads and socket ownership
      3. 20.3 Request-reply, publish-subscribe and pipelines
        1. 20.3.1 Routing envelopes and ROUTER identities
      4. 20.4 Lost replies and dropped publications
      5. 20.5 Queue limits, reconnection and shutdown
      6. 20.6 Polling several sockets and using asyncio
      7. 20.7 Monitoring socket connections
      8. 20.8 Security and cloud networks
      9. 20.9 Web services and background workers
        1. 20.9.1 Early and late acknowledgment in Celery
        2. 20.9.2 Retries, poison tasks and revocation
        3. 20.9.3 Routing tasks and choosing worker pools
      10. 20.10 Testing publish and subscribe
      11. 20.11 Choosing between ZeroMQ and a durable broker
      12. 20.12 Distributed compute with Ray
        1. 20.12.1 Ray tasks and stateful actors
        2. 20.12.2 Removing duplicate tasks with a shared ledger
        3. 20.12.3 Object references and cloudpickle serialization
        4. 20.12.4 Scheduling with resource labels and placement groups
        5. 20.12.5 Backpressure by limiting pending tasks
        6. 20.12.6 Worker failure, actor restart and driver loss
        7. 20.12.7 Use cases for Ray
      13. 20.13 Verifying delivery semantics and worker limits
    6. Chapter 21 WebSockets
      1. 21.1 The HTTP upgrade to WebSocket
        1. 21.1.1 Checking the origin and authenticating browsers
      2. 21.2 Data frames, control frames and message limits
        1. 21.2.1 The WebSocket close handshake
      3. 21.3 Heartbeats, ping-pong frames and idle timeouts
        1. 21.3.1 Keepalive pings and event acknowledgments
      4. 21.4 Routing events to the instance that holds a connection
      5. 21.5 Managing the tasks for one connection
      6. 21.6 Reconnecting clients and replaying events
      7. 21.7 Planning WebSocket capacity
      8. 21.8 GraphQL fundamentals
        1. 21.8.1 Comparing REST and GraphQL
        2. 21.8.2 Resolvers and authorization
        3. 21.8.3 The N plus one query problem
        4. 21.8.4 Limiting GraphQL pagination, depth and cost
        5. 21.8.5 Changing a GraphQL schema safely
      9. 21.9 GraphQL subscriptions over WebSocket
      10. 21.10 Testing GraphQL over WebSocket
      11. 21.11 Choosing a transport for the command-line client
      12. 21.12 Verifying real-time subscription and replay behavior
  5. Part V Coordination

    1. Chapter 22 Time and ordering
      1. 22.1 Wall clocks and monotonic clocks
      2. 22.2 Wall-clock accuracy and error bounds
      3. 22.3 Causality and the happens-before relation
      4. 22.4 Lamport clocks
      5. 22.5 Vector clocks
      6. 22.6 Hybrid logical clocks
      7. 22.7 Keeping logical clocks correct across restarts
      8. 22.8 Working with time in Python
        1. 22.8.1 Timezone-aware dates with Pendulum
      9. 22.9 Diagnosing ordering problems and testing with fake clocks
    2. Chapter 23 Consensus
      1. 23.1 The primary election problem
      2. 23.2 Quorums and the split-brain hazard
      3. 23.3 The four properties of consensus
        1. 23.3.1 Assumptions about messages, restarts and progress
      4. 23.4 Choosing between consensus and an existing coordination service
      5. 23.5 Failure models
        1. 23.5.1 Detecting failures with timeouts
      6. 23.6 Limits on progress and availability
        1. 23.6.1 The FLP impossibility result
        2. 23.6.2 The CAP theorem and network partitions
      7. 23.7 Majority quorums
        1. 23.7.1 Quorum overlap guarantees
      8. 23.8 Handling writes during a network partition
      9. 23.9 Changing cluster membership
      10. 23.10 Crash faults and Byzantine faults
      11. 23.11 Comparing Paxos, Raft, Zab and PBFT
      12. 23.12 Alternatives to application-owned consensus
      13. 23.13 Building a quorum register in the lab
    3. Chapter 24 Consensus algorithms
      1. 24.1 Raft and Paxos
      2. 24.2 Raft roles and node state
      3. 24.3 Terms in Raft
      4. 24.4 Raft leader election
      5. 24.5 Raft log replication and commitment
      6. 24.6 Persisting terms, votes and log entries
        1. 24.6.1 Crashes during a write
      7. 24.7 Snapshots and membership changes
      8. 24.8 Client retries and deduplication
      9. 24.9 Monitoring and diagnosing a Raft cluster
      10. 24.10 Testing elections, crashes and partitions
      11. 24.11 Choosing a coordination service
      12. 24.12 Building the election lab
    4. Chapter 25 Replication and state machines
      1. 25.1 The replicated state machine
        1. 25.1.1 Designing replicated commands
      2. 25.2 Committing, applying and replying to clients
        1. 25.2.1 Detecting duplicate client requests
      3. 25.3 Side effects and the outbox pattern
      4. 25.4 Repairing a follower's log
      5. 25.5 Snapshots and log compaction
      6. 25.6 Committed commands that fail to apply
      7. 25.7 Read freshness and stale-read mitigation
        1. 25.7.1 Linearizability requirements
      8. 25.8 Persistence and crash consistency
      9. 25.9 The cost of replication
    5. Chapter 26 Distributed locking
      1. 26.1 Locking for efficiency and locking for correctness
      2. 26.2 Atomic command execution in Redis
        1. 26.2.1 Lost updates with GET and SET
        2. 26.2.2 Expiring key leases with SET NX PX
        3. 26.2.3 Checking lock ownership with Lua scripts
        4. 26.2.4 Limits of atomic commands
      3. 26.3 Writes after lease expiry
        1. 26.3.1 Renewing a lease
      4. 26.4 Fencing tokens and stale-write rejection
      5. 26.5 Conditional writes in Azure and on-prem storage
      6. 26.6 Stale-write rejection at the protected resource
      7. 26.7 Message broker locks
      8. 26.8 Monitoring and debugging locks
        1. 26.8.1 Handling acquire, renew and release failures
      9. 26.9 Choosing a coordination mechanism
  6. Part VI State and storage

    1. Chapter 27 Stateless services
      1. 27.1 State in a stateless service
      2. 27.2 Handling retries across instances
      3. 27.3 Loading configuration and secrets at startup
        1. 27.3.1 Managed identities and how long to keep secrets
      4. 27.4 ASGI applications and worker processes
      5. 27.5 Liveness and readiness probes
      6. 27.6 Graceful shutdown
      7. 27.7 Counters, schedulers and updates across instances
      8. 27.8 Retrying and limiting load
      9. 27.9 Validating multi-instance state sharing and connection draining
    2. Chapter 28 Distributed data storage
      1. 28.1 Partitioning keys across nodes
        1. 28.1.1 Placing replicas in separate failure domains
      2. 28.2 Choosing a partition key
      3. 28.3 Rebalancing data while the store is running
      4. 28.4 Single-leader, multi-leader and leaderless replication
      5. 28.5 Repairing replicas
        1. 28.5.1 Deleting data with tombstones
      6. 28.6 Consistency models
      7. 28.7 Resolving concurrent writes
      8. 28.8 Partition limits and hot partitions in Azure
        1. 28.8.1 Diagnosing a hot partition in Cosmos DB
      9. 28.9 Partitioning and placement on-prem
      10. 28.10 Validating ring stability and replica repair
    3. Chapter 29 Key-value stores
      1. 29.1 Redis data structures and commands
        1. 29.1.1 Python pipeline transactions and optimistic concurrency with WATCH
        2. 29.1.2 Persistence, eviction, key scans and cluster slots
        3. 29.1.3 Persistence files and replication
        4. 29.1.4 Choosing an eviction policy
      2. 29.2 Refilling a cache and keeping it fresh
      3. 29.3 Cosmos DB items and consistency levels
        1. 29.3.1 Request-unit charges and throttling
        2. 29.3.2 Choosing a partition key and an indexing policy
        3. 29.3.3 Retries and the change feed
        4. 29.3.4 Changing a partition key and logical partition size limits
      4. 29.4 Choosing between a cache and an authoritative store
        1. 29.4.1 Migrating to Azure Managed Redis
      5. 29.5 Valkey and PostgreSQL on-prem
      6. 29.6 Validating key-value storage semantics
  7. Part VII Orchestration

    1. Chapter 30 Directed acyclic graphs (DAGs)
      1. 30.1 Validating a graph before it runs
      2. 30.2 Topological ordering and cycle detection
      3. 30.3 Running ready nodes on several workers
      4. 30.4 Deterministic scheduling
      5. 30.5 Estimating run time with the critical path
      6. 30.6 Graph levels and redundant edges
      7. 30.7 Failure policies
      8. 30.8 Attempts and retries
      9. 30.9 Storing run state in a database
      10. 30.10 Resuming a run after a crash
      11. 30.11 Diagnosing a stuck or failed run
      12. 30.12 Deriving edges from inputs and outputs
    2. Chapter 31 Workflow orchestration
      1. 31.1 Airflow components and the metadata database
      2. 31.2 Keeping DAG files fast to import
        1. 31.2.1 Slow imports and delayed runs
      3. 31.3 Data intervals and logical dates
      4. 31.4 Planning capacity for catchup and backfill
      5. 31.5 Retry-safe and replay-safe tasks
      6. 31.6 Passing results between tasks with XCom
      7. 31.7 Sensors and deferrable operators
      8. 31.8 Concurrency controls
      9. 31.9 Managing secrets and identities
      10. 31.10 Deploying and upgrading Airflow
      11. 31.11 Running Airflow on Azure
      12. 31.12 Running Airflow on-prem
      13. 31.13 Alternatives to Airflow
  8. Part VIII Systems operations

    1. Chapter 32 Containers and Kubernetes
      1. 32.1 Process isolation options
      2. 32.2 Linux security controls for containers
      3. 32.3 From OCI image to running process
      4. 32.4 Running virtual machines with QEMU and KVM
      5. 32.5 Managing virtual machines with libvirt
      6. 32.6 Controlling a running VM with QMP
      7. 32.7 Application containers and system containers
      8. 32.8 Kata Containers and the utility VM
      9. 32.9 The local lab platform
      10. 32.10 The SigRaft on-prem deployment
      11. 32.11 Building application container images
      12. 32.12 Registries, tags and digests
      13. 32.13 Azure Container Apps and AKS
      14. 32.14 Kubernetes objects for a stateless service
      15. 32.15 The Kubernetes Python client
      16. 32.16 Kubernetes pod scheduling
      17. 32.17 Workload identity in a container
      18. 32.18 Rolling updates and the limits of rollback
      19. 32.19 Moving from a local kind cluster to AKS
      20. 32.20 Routing requests through Nginx
      21. 32.21 Health probes, autoscaling and disruption budgets
      22. 32.22 Namespace isolation and coexisting versions in Kubernetes
        1. 32.22.1 Kubernetes rolling updates
        2. 32.22.2 Canary traffic splitting in Kubernetes
        3. 32.22.3 Blue-green environment switching in Kubernetes
    2. Chapter 33 Failure and backpressure
      1. 33.1 Request deadlines and timeout budget propagation
      2. 33.2 Exponential backoff and jittered retry policies
        1. 33.2.1 Backoff, jitter and retry limits
      3. 33.3 Retry amplification and overload collapse
      4. 33.4 Circuit breaker states and trip thresholds
      5. 33.5 Resource pool isolation with bulkheads
      6. 33.6 Sizing a queue
      7. 33.7 Admission control and load shedding
      8. 33.8 Health checks during overload
      9. 33.9 Detecting and diagnosing service overload
      10. 33.10 Testing deadlines and overload
    3. Chapter 34 Observability
      1. 34.1 Choosing what telemetry to collect
      2. 34.2 Structured logging
      3. 34.3 Request IDs and trace IDs
      4. 34.4 Metrics and latency distributions
      5. 34.5 Traces and spans
      6. 34.6 OpenTelemetry providers, exporters and the collector
        1. 34.6.1 Baggage and passing context between threads and processes
      7. 34.7 Azure Monitor and Application Insights
      8. 34.8 Health checks and service-level health
      9. 34.9 Service level objectives, error budgets and alerts
      10. 34.10 Tracing an incident
    4. Chapter 35 Operational data analysis
      1. 35.1 Defining the decision, the population and the observation
      2. 35.2 Arranging data in tidy tables
      3. 35.3 NumPy arrays, masks and default settings
      4. 35.4 Loading and validating data with pandas
      5. 35.5 Describing a distribution
      6. 35.6 Confidence intervals and comparing releases
      7. 35.7 Regression on job duration
      8. 35.8 Accessible charts and tables
      9. 35.9 Sampling errors and measurement errors
      10. 35.10 Effect size, uncertainty and release decisions
  9. Part IX The SigRaft on-prem cloud

    1. Chapter 36 SigRaft on-prem cloud infrastructure
      1. 36.1 OpenStack and Kubernetes responsibilities
      2. 36.2 Validating the configuration and planning changes
      3. 36.3 Sizing the host and its virtual machines
      4. 36.4 Separating networks and naming services with DNS
      5. 36.5 Storage services and backups
      6. 36.6 Managing identities and secrets
      7. 36.7 Tracing requests and monitoring the platform
      8. 36.8 Failure domains, deployment order and teardown
      9. 36.9 Bridging private infrastructure components to Azure equivalents
    2. Chapter 37 On-prem cloud APIs with Python
      1. 37.1 Adapter contracts bridging private cloud APIs and Azure SDKs
      2. 37.2 OpenStackSDK service proxies
      3. 37.3 Kubernetes desired state and rollout readiness
      4. 37.4 Harbor HTTP clients
      5. 37.5 Managing rack servers with Redfish
      6. 37.6 Per-adapter identities
      7. 37.7 Pagination and resumable reads
      8. 37.8 Operations, readiness and deadlines
      9. 37.9 Lost requests and lost responses
      10. 37.10 Planning and applying resource changes
      11. 37.11 Client lifetime and logging
      12. 37.12 Fake clients and tests against live platforms
  10. Part X Resource scheduling

    1. Chapter 38 Resource-aware job scheduling
      1. 38.1 Queues, workers and schedulers
      2. 38.2 Describing a job's resource requirements
      3. 38.3 Tracking nodes with inventory reports and heartbeats
      4. 38.4 Rejecting impossible requests and queuing waiting ones
      5. 38.5 Choosing a node with filtering and scoring
      6. 38.6 Electing one scheduler and storing reservations
      7. 38.7 Sending allocations to node queues
      8. 38.8 Enforcing resource limits on the node
        1. 38.8.1 Limiting CPU and memory with cgroups
        2. 38.8.2 Keeping CPUs, GPUs and memory on one NUMA node
        3. 38.8.3 Limiting which GPUs a job can see
      9. 38.9 Job states, allocation leases and node failure
      10. 38.10 Priority, FIFO order and project quotas
      11. 38.11 Advanced scheduling policies
      12. 38.12 Comparing Slurm, Kubernetes, Ray and Airflow
      13. 38.13 Running the scheduler locally, on-prem and on Azure
      14. 38.14 Logging decisions and securing the scheduler
  11. Part XI Release engineering

    1. Chapter 39 The completed SigRaft service
      1. 39.1 Promoting one image by digest
      2. 39.2 REST and GraphQL interfaces
      3. 39.3 WebSocket commands and job notifications
        1. 39.3.1 Authenticating and authorizing WebSocket clients
        2. 39.3.2 Initial snapshots, event ordering and slow clients
      4. 39.4 Versioning the service documentation
      5. 39.5 Reserving resources before dispatching work
      6. 39.6 Release quality reports
        1. 39.6.1 Releasing to the on-prem cloud
        2. 39.6.2 Release reports for Azure
      7. 39.7 Azure pipelines, identities and approvals
      8. 39.8 Configuring each environment
        1. 39.8.1 Reloading configuration without a restart
      9. 39.9 Multi-strategy deployment pipelines for SigRaft
      10. 39.10 Traffic draining and zero-downtime service transitions
      11. 39.11 Configuring hosts with Ansible
      12. 39.12 Running short remote tasks with Fabric
      13. 39.13 Validating a deployment with telemetry
        1. 39.13.1 Read-only hardware inventory
      14. 39.14 Diagnosing and recovering from a failed rollout

Appendices

  1. Appendix A The lab toolchain
    1. A.1 Installing and pinning the toolchain
    2. A.2 Inspecting lab code at the Python prompt
    3. A.3 Commands for creating projects and running gates
    4. A.4 Exit statuses
    5. A.5 Configuring a lab project
    6. A.6 Configuration precedence
    7. A.7 The tools behind each gate
    8. A.8 Skipped gates
    9. A.9 Publishing results as JUnit XML
    10. A.10 Running the gates offline
    11. A.11 Running every lab's quality gates
  2. Appendix B Linux command-line tutorial and reference
    1. B.1 The shell, terminal and working directory
    2. B.2 Creating, copying, moving and removing files
    3. B.3 Reading and writing text
    4. B.4 Redirection and pipelines
    5. B.5 Finding files and matching text
    6. B.6 Command history
    7. B.7 Environment variables, processes and jobs
    8. B.8 Connecting with SSH
    9. B.9 A first Bash script
    10. B.10 Editing files with Vim
    11. B.11 Diagnostic commands
      1. B.11.1 Sockets, connections and processes
      2. B.11.2 Name resolution and IP routing
      3. B.11.3 Firewall rules and packet capture
      4. B.11.4 Tracing system calls and Python stacks
      5. B.11.5 Cgroup limits and out-of-memory kills
      6. B.11.6 Inspecting a systemd service
      7. B.11.7 Simulating network delay, loss and reordering
      8. B.11.8 Signals and graceful shutdown
      9. B.11.9 Kernel network settings
      10. B.11.10 Permissions for diagnostic commands
      11. B.11.11 Cleaning up after diagnosis
  3. Appendix C Git development flow tutorial
    1. C.1 Configuring Git and getting help
    2. C.2 The working tree, staging area and first commit
    3. C.3 Ignoring generated and private files
    4. C.4 Reading history and comparing revisions
    5. C.5 Branches and HEAD
    6. C.6 Working with remote repositories
    7. C.7 Choosing a branching method
    8. C.8 Merging branches
    9. C.9 Resolving merge conflicts
    10. C.10 Rebasing a private branch
    11. C.11 Cherry-picking commits
    12. C.12 Creating pull requests from the command line
    13. C.13 Submodules
    14. C.14 Undoing changes and checking repository status

References

Reader Resources

Companion Labs

Use the public lab repository for the runnable chapter exercises, setup instructions and quality checks. The companion software is licensed separately from the book.

Open the Lab Repository

Corrections and Enquiries

For a correction, include the edition, chapter or section, and enough detail to locate the issue. Publication and permissions enquiries use the same address.

Contact Publications

Edition and Availability

First edition, 2026. Amazon paperback ISBN: 9798178152348. Imprint: independently published.

LuLu paperback ISBN: 978-1-291-48183-9.

A paperback edition is available directly from LuLu. Kindle and paperback editions are also available through Amazon. Use the region selector above; availability and pricing may vary between Amazon marketplaces.

The book is copyright 2026 Weqaar Janjua, all rights reserved. Its private manuscript and full PDF are not distributed on this page. The public companion software licence does not grant permission to reproduce or redistribute the book.

Back to Books

Cite as

Cite this edition using the BibTeX entry below. The other citation styles are generated from the same book record.

@book{janjua2026linuxdistributedpython,
  author = {Janjua, Weqaar},
  title = {Linux Distributed Systems Programming with {Python}: A Lab-Driven Systems-Level Guide to Building Reliable Distributed Systems},
  year = {2026},
  edition = {1},
  publisher = {Independently published},
  isbn = {9798178152348},
  url = {https://packetfive.com/0x0001}
}