Table of Contents
First edition · 39 chapters in 11 parts, plus appendices. Expand a chapter to see its sections and subsections.
Contents headings only; the full text is not available on this website. Page numbers are omitted.
Preface
- About this book
- Terms used in this book
- Intended readership
- Book organization
- Ethics and AI-assisted development
- Acknowledgments
-
Part I Production engineering
-
Chapter 1 The engineering process
-
1.1 Command-line and version-control prerequisites
-
1.2 Environment drift
-
1.3 Continuous integration and quality gates
- 1.3.1 Builds and artifacts
- 1.3.2 Pipelines, jobs and steps
- 1.3.3 Workers, agents and nodes
- 1.3.4 Formatting, linting, types and tests
- 1.3.5 Stages of a CI run
- 1.3.6 Hidden failures in passing builds
-
1.4 The development environment
- 1.4.1 Installing Visual Studio Code
- 1.4.2 Installing the extensions
-
1.5 Software testing
- 1.5.1 Unit tests
- 1.5.2 Functional tests
- 1.5.3 Confirming a fix with a failing test
-
1.6 The coding standard
- 1.6.1 Classes, functions and interfaces
- 1.6.2 Type hints and the Any type
- 1.6.3 Error handling and resource cleanup
-
1.7 AI coding agents
-
1.8 Local and privacy-preserving agents
- 1.8.1 A vendor-neutral local setup
- 1.8.2 Local inference hardware
- 1.8.3 Prompt data flow
- 1.8.4 Verifying the LLM file and its license
- 1.8.5 Context leakage and resource limits
- 1.8.6 Accepting changes from a local agent
-
1.9 Reading gate results
-
1.10 Building reproducible environments
-
1.11 Installing the toolchain
-
1.12 Configuring the gates in pyproject.toml
-
1.13 Distinguishing code violations from tool crashes
-
1.14 Code coverage as an acceptance gate
-
1.15 Automating gate execution in continuous integration
-
1.16 Archiving quality reports and build artifacts
-
1.17 Jobs and job status
-
Chapter 2 Packaging and building a Python project
-
2.1 Project templates
-
2.2 Source distributions and wheels
-
2.3 Wheel contents and compatibility tags
-
2.4 Binary compatibility and manylinux wheels
-
2.5 Standalone executables
-
2.6 Bundling with PyInstaller and compiling with Nuitka
-
2.7 Building on each target operating system
-
2.8 Defining a build matrix
-
2.9 Reading an ELF executable
-
2.10 Windows Portable Executable files
-
2.11 Project metadata and build configuration
-
2.12 The src layout and testing the installed package
-
2.13 Version ranges and pinned versions
-
2.14 Reproducible builds, SBOMs and signatures
-
2.15 Publishing to a private package index
-
2.16 Building documentation with Sphinx
- 2.16.1 Writing docstrings and comments
- 2.16.2 Previewing the documentation site
- 2.16.3 Comparing Sphinx with MkDocs
-
Chapter 3 Code quality and compliance
-
3.1 Architecture of a quality gate suite
-
3.2 Interpreting tool exit codes with adapters
-
3.3 Unified gate execution across local workstations and CI
-
3.4 Reporting results in JUnit XML
-
3.5 Enforcing coverage thresholds with pytest-cov
-
3.6 Writing tests with pytest and unittest
-
3.7 Mocks, fakes and test doubles
-
3.8 Auditing dependencies for vulnerabilities and licenses
-
3.9 Suppressing false positives
-
3.10 Publishing quality reports in Azure Pipelines
-
3.11 Formatting, docstrings and type annotations
-
3.12 Testing the documentation build
-
Chapter 4 Command line tools
-
4.1 Separating offline, service and cloud commands
-
4.2 Standard streams and interactive terminals
-
4.3 Output stream buffering in standard streams
- 4.3.1 Pipe capacity and backpressure
-
4.4 Defining subcommands with argparse
-
4.5 Building commands with Typer
-
4.6 Configuration source precedence
-
4.7 CLI command exit codes and error status reporting
- 4.7.1 Handling Ctrl-C and termination signals
-
4.8 Human-readable and machine-readable output
-
4.9 Authenticating with bearer tokens
-
4.10 Calling the job API over HTTP
-
4.11 Retries, backoff and command deadlines
-
4.12 Shell completion and help
-
4.13 Testing commands without a network
-
Chapter 5 Object-oriented design in Python
-
5.1 The Python object model and memory attributes
- 5.1.1 Methods, functions and self
- 5.1.2 Special methods and Python's built-in operations
- 5.1.3 Inspecting objects in the interpreter
-
5.2 Names, identity and aliasing
-
5.3 Defining domain values with dataclasses
-
5.4 Encapsulation and private attributes
- 5.4.1 Properties and descriptors
-
5.5 Interfaces with protocols and abstract base classes
-
5.6 Composition and dependency injection
-
5.7 Inheritance, polymorphism and method resolution order
-
5.8 Choosing between lists and dictionaries
-
5.9 Function and method decorators
-
5.10 Reflection and metaclasses
-
5.11 Organizing relay into layers of objects
-
Chapter 6 Trees and graphs for distributed systems
-
6.1 Measuring operation cost with Big-O notation
-
6.2 Trees and ordered indexes
- 6.2.1 Rooted trees
- 6.2.2 Tree traversal
- 6.2.3 Updating and removing nodes
- 6.2.4 Common tree structures
- 6.2.5 Deleting from a binary search tree
- 6.2.6 Rebuilding and self-balancing trees
- 6.2.7 Balancing a red-black tree
-
6.3 Representing graphs in Python
- 6.3.1 Graph terms and changing a graph
- 6.3.2 The graphlib module and graph interfaces
-
6.4 Depth-first and breadth-first graph search
-
6.5 Shortest paths on weighted graphs
- 6.5.1 Dijkstra's algorithm and edge relaxation
- 6.5.2 Bellman-Ford and negative cycles
-
6.6 OSPF route calculation
-
6.7 Dependency graphs and topological sorting
-
6.8 Trees and graphs in distributed systems
-
6.9 Testing correctness before optimizing
-
Chapter 7 Debugging, profiling and benchmarking
-
7.1 Defining the problem
- 7.1.1 Overhead, permissions and sensitive data
-
7.2 Debugging Python code with pdb
- 7.2.1 Reproducing the failure first
- 7.2.2 Starting the debugger from code and after a crash
- 7.2.3 Inspecting live objects
- 7.2.4 The dir function and the __debug__ constant
-
7.3 Timing one operation
- 7.3.1 Choosing a clock
- 7.3.2 Timing with timeit
-
7.4 Profiling Python code
- 7.4.1 Profiling every call with cProfile
- 7.4.2 Choosing what to profile and the cost of profiling
- 7.4.3 Tracing memory allocation with tracemalloc
-
7.5 Benchmarking with pyperf
- 7.5.1 Defining what the benchmark measures
- 7.5.2 Writing benchmarks with the pyperf Runner
- 7.5.3 Repeated measurements in pyperf
- 7.5.4 Running and saving benchmarks
- 7.5.5 Preparing the benchmark machine
- 7.5.6 Reading the spread of benchmark results
- 7.5.7 Comparing a baseline with a candidate
- 7.5.8 Common pyperf mistakes
-
7.6 Debugging native code with GDB
- 7.6.1 Building with debug symbols
- 7.6.2 Attaching to a process and reading core dumps
-
7.7 Tracing system calls with strace
-
7.8 Inspecting open files, sockets and process state
-
7.9 Counting events and sampling with perf
-
7.10 Tracing kernel events with ftrace
-
7.11 Investigating a worker blocked on I/O
-
Part II The Azure platform
-
Chapter 8 Azure infrastructure for distributed systems
-
8.1 IaaS, PaaS and SaaS cloud service models
- 8.1.1 Service model selection for SigRaft
-
8.2 Comparing Azure cloud services to on-premises counterparts
-
8.3 Azure resource scopes and inherited policy
-
8.4 Managing resources with Azure Resource Manager
-
8.5 Identities and role-based access control
-
8.6 Control-plane and data-plane permissions
-
8.7 Regions and availability zones
-
8.8 Virtual networks, private endpoints and DNS
-
8.9 Controlling cost and cleaning up resources
-
8.10 Diagnosing a failed deployment
-
8.11 Declarative infrastructure with Bicep
- 8.11.1 Why we use Bicep
- 8.11.2 How we use Bicep
- 8.11.3 Bicep in the SigRaft project and labs
-
8.12 Describing SigRaft infrastructure in Bicep
- 8.12.1 Parameters and parameter files
- 8.12.2 Modules, dependencies and deployment scopes
-
8.13 Containers, Kubernetes and workload identity
- 8.13.1 Containers and operating-system virtualization
- 8.13.2 Types of container technologies
- 8.13.3 Kubernetes and container orchestration
- 8.13.4 Container registries and workload identity in Azure
- 8.13.5 Container and Kubernetes coverage across the labs
-
8.14 Deploying and tearing down infrastructure
-
8.15 Simulating cloud deployments with local resource graphs
-
Chapter 9 The Azure Python SDKs
-
9.1 Management and data-plane packages
-
9.2 Portable client adapter design across cloud and on-prem backends
-
9.3 Choosing credentials for each deployment
-
9.4 Creating and closing clients
-
9.5 Reading paged results
-
9.6 Long-running operations
-
9.7 Retries and idempotent operations
-
9.8 Handling request and response failures
-
9.9 Asynchronous clients
-
9.10 Logging SDK calls without leaking secrets
-
9.11 Testing adapters with protocols and fakes
-
Part III Python and the operating system
-
Chapter 10 Python program execution
-
10.1 From source text to a running process
-
10.2 The Python language, implementations and packages
-
10.3 Starting the Python interpreter on Linux
-
10.4 From source text to a syntax tree
- 10.4.1 Tokenizing source code
- 10.4.2 Parsing tokens into a syntax tree
- 10.4.3 Traversing and transforming the syntax tree
-
10.5 Processor instructions and virtual-machine instructions
- 10.5.1 Encoding integer addition in RISC-V
- 10.5.2 Implementing ADD with digital logic
- 10.5.3 Instructions, opcodes and operands
- 10.5.4 Stacks and stack machines
-
10.6 Bytecode execution in a virtual machine
-
10.7 Reading CPython bytecode
- 10.7.1 Adaptive specialization and JIT compilation
-
10.8 Frames, objects and function calls
-
10.9 Portability across CPU architectures
-
10.10 Python on x86-64 Linux
-
10.11 C extensions and Cython
-
10.12 Inspecting source, bytecode and native modules
-
Chapter 11 Processes, concurrency and NUMA performance
-
11.1 Linux thread scheduling and container CPU limits
-
11.2 Concurrency versus parallelism
- 11.2.1 Multiprocessing versus asynchronous execution
- 11.2.2 Native threads versus green threads
- 11.2.3 Concurrency and parallelism across the labs
-
11.3 Processes, threads and the GIL
-
11.4 Worker process communication
- 11.4.1 Process start methods and lifetimes
- 11.4.2 Queue, SimpleQueue and JoinableQueue
- 11.4.3 Managers and shared proxies
- 11.4.4 Process pools, result ordering and submission limits
-
11.5 Sharing large data through shared memory
-
11.6 Multicore performance and memory locality
-
11.7 NUMA locality on Linux
- 11.7.1 NUMA page placement
- 11.7.2 CPU affinity and memory policy
- 11.7.3 Python libnuma bindings
- 11.7.4 Balancing process workers across nodes
- 11.7.5 Measuring locality and cache behavior
-
11.8 GPU worker placement
- 11.8.1 GPU runtimes and drivers
- 11.8.2 PCIe topology and NUMA
- 11.8.3 Host-pinned and device memory
- 11.8.4 DMA transfers and IOMMU device isolation
-
11.9 Remote direct memory access for accelerators
- 11.9.1 GPUDirect RDMA requirements
- 11.9.2 InfiniBand and RoCE network transports
- 11.9.3 RoCEv2 congestion control
- 11.9.4 Discovering accelerator devices from sysfs
- 11.9.5 Requesting accelerators in Kubernetes
-
11.10 Sizing worker pools for blocking calls
-
11.11 Asyncio and cooperative scheduling
-
11.12 Choosing threads, processes or asyncio
-
11.13 Signals and graceful shutdown
-
Chapter 12 Sockets and the network stack
-
12.1 TCP streams and UDP datagrams in Python
- 12.1.1 Python sockets and Linux system calls
- 12.1.2 Packet loss in TCP and UDP
-
12.2 From application message to Ethernet frame
- 12.2.1 Receiving and decapsulating a frame
-
12.3 Linux packet buffers and sk_buff
- 12.3.1 Linked lists and packet queues
- 12.3.2 Sending through a net_device
- 12.3.3 Receiving from a device
-
12.4 Building packets offline with Scapy
-
12.5 Name resolution and DNS
-
12.6 Creating, binding and connecting sockets
-
12.7 The TCP handshake and accept queues on Linux
-
12.8 TCP as a byte stream
-
12.9 Partial socket reads and writes
-
12.10 Waiting for many sockets with epoll
-
12.11 TCP retransmission, flow control and congestion control
-
12.12 Closing TCP connections, resets and TIME_WAIT
-
12.13 Diagnosing DNS, routing and connection problems
-
12.14 Testing sockets on loopback
-
Chapter 13 Packet-level network programming
-
13.1 Network layers and packet inspection tools
-
13.2 Capture location and offloads
-
13.3 Packet headers
-
13.4 Checksums and offload
-
13.5 MTU, segmentation and fragmentation
-
13.6 TCP probes and captures with Scapy
-
13.7 Event filtering and counting with eBPF
-
13.8 Privileges and isolation
-
13.9 Packet loss and connection diagnostics
-
13.10 Network fault injection with netem
-
Chapter 14 Cython and systems-level integration
-
14.1 Profiling before choosing native code
-
14.2 Compiling a Cython extension
-
14.3 The CPython C API and binary compatibility
-
14.4 Releasing the GIL for native work
-
14.5 C types and memoryviews in a checksum loop
-
14.6 Calling Linux C functions and handling errors
-
14.7 Memory ownership and error paths
-
14.8 Debugging native crashes
-
14.9 Platform-specific extension wheels
-
14.10 Choosing Cython, ctypes, cffi or Rust bindings
-
Chapter 15 Engineering CPython bytecode
-
15.1 Parsing Python source
- 15.1.1 Writing a query language with PyParsing
-
15.2 Name resolution and bytecode generation in CPython
-
15.3 CPython stack operations and instruction encoding
-
15.4 Instruction execution in a CPython frame
-
15.5 Connecting a Python function to an opcode
-
15.6 Patching a pinned CPython release
-
15.7 Testing an interpreter change
-
15.8 Processor architecture and distributing bytecode
-
Part IV Talking between processes
-
Chapter 16 Protocols and framing
-
16.1 Message boundaries in a TCP stream
-
16.2 Choosing a framing method
-
16.3 Delimiter-based framing and escaping
-
16.4 Fixed-width framing
-
16.5 Length-prefixed framing
-
16.6 Parsing frames incrementally
- 16.6.1 Fixed-size ring buffers
-
16.7 Handling end of stream
-
16.8 Rejecting bad frames and setting read timeouts
-
16.9 Framing with asyncio streams
-
16.10 Versioning a protocol
-
16.11 Testing a frame parser
-
16.12 Diagnosing framing problems
-
Chapter 17 Serialization and wire formats
-
17.1 Wire format requirements
-
17.2 Encoding Python values as JSON
-
17.3 Encoding with MessagePack
-
17.4 Protocol Buffers schemas and field numbers
-
17.5 Protocol Buffers field encoding
-
17.6 Field presence and default values in Protocol Buffers
-
17.7 Schema evolution
-
17.8 Reviewing schemas and testing compatibility
-
17.9 Choosing between JSON and Protocol Buffers
-
17.10 Decoding untrusted messages safely
-
Chapter 18 Data validation
-
18.1 Validation boundaries in service architectures
-
18.2 Validating input with Pydantic
-
18.3 Defining a job submission model
-
18.4 Strict mode and type coercion
-
18.5 Reporting validation errors without leaking data
-
18.6 Handling unknown fields
-
18.7 Choosing a command model with a discriminator field
-
18.8 Changing validation models over time
-
18.9 Validating timestamps, money, identifiers and URLs
-
18.10 Limiting the size of untrusted input
-
18.11 Parsing throughput and validation overhead
-
18.12 Validating configuration at startup
-
Chapter 19 Remote procedure calls
-
19.1 Local and remote calls
-
19.2 Designing HTTP resources and status codes
-
19.3 Building an HTTP service with FastAPI
-
19.4 Running the service with Uvicorn and uvloop
-
19.5 gRPC over HTTP/2
-
19.6 Timeouts and shared deadlines
-
19.7 Transport errors and application errors
-
19.8 Retry decisions for failed calls
-
19.9 Idempotency keys and stored results
-
19.10 Streaming and backpressure
-
19.11 Versioning, authentication and logging
-
19.12 Testing RPC failures
-
Chapter 20 Message patterns
-
20.1 Message framing and connections in ZeroMQ
-
20.2 Contexts, I/O threads and socket ownership
-
20.3 Request-reply, publish-subscribe and pipelines
- 20.3.1 Routing envelopes and ROUTER identities
-
20.4 Lost replies and dropped publications
-
20.5 Queue limits, reconnection and shutdown
-
20.6 Polling several sockets and using asyncio
-
20.7 Monitoring socket connections
-
20.8 Security and cloud networks
-
20.9 Web services and background workers
- 20.9.1 Early and late acknowledgment in Celery
- 20.9.2 Retries, poison tasks and revocation
- 20.9.3 Routing tasks and choosing worker pools
-
20.10 Testing publish and subscribe
-
20.11 Choosing between ZeroMQ and a durable broker
-
20.12 Distributed compute with Ray
- 20.12.1 Ray tasks and stateful actors
- 20.12.2 Removing duplicate tasks with a shared ledger
- 20.12.3 Object references and cloudpickle serialization
- 20.12.4 Scheduling with resource labels and placement groups
- 20.12.5 Backpressure by limiting pending tasks
- 20.12.6 Worker failure, actor restart and driver loss
- 20.12.7 Use cases for Ray
-
20.13 Verifying delivery semantics and worker limits
-
Chapter 21 WebSockets
-
21.1 The HTTP upgrade to WebSocket
- 21.1.1 Checking the origin and authenticating browsers
-
21.2 Data frames, control frames and message limits
- 21.2.1 The WebSocket close handshake
-
21.3 Heartbeats, ping-pong frames and idle timeouts
- 21.3.1 Keepalive pings and event acknowledgments
-
21.4 Routing events to the instance that holds a connection
-
21.5 Managing the tasks for one connection
-
21.6 Reconnecting clients and replaying events
-
21.7 Planning WebSocket capacity
-
21.8 GraphQL fundamentals
- 21.8.1 Comparing REST and GraphQL
- 21.8.2 Resolvers and authorization
- 21.8.3 The N plus one query problem
- 21.8.4 Limiting GraphQL pagination, depth and cost
- 21.8.5 Changing a GraphQL schema safely
-
21.9 GraphQL subscriptions over WebSocket
-
21.10 Testing GraphQL over WebSocket
-
21.11 Choosing a transport for the command-line client
-
21.12 Verifying real-time subscription and replay behavior
-
Part V Coordination
-
Chapter 22 Time and ordering
-
22.1 Wall clocks and monotonic clocks
-
22.2 Wall-clock accuracy and error bounds
-
22.3 Causality and the happens-before relation
-
22.4 Lamport clocks
-
22.5 Vector clocks
-
22.6 Hybrid logical clocks
-
22.7 Keeping logical clocks correct across restarts
-
22.8 Working with time in Python
- 22.8.1 Timezone-aware dates with Pendulum
-
22.9 Diagnosing ordering problems and testing with fake clocks
-
Chapter 23 Consensus
-
23.1 The primary election problem
-
23.2 Quorums and the split-brain hazard
-
23.3 The four properties of consensus
- 23.3.1 Assumptions about messages, restarts and progress
-
23.4 Choosing between consensus and an existing coordination service
-
23.5 Failure models
- 23.5.1 Detecting failures with timeouts
-
23.6 Limits on progress and availability
- 23.6.1 The FLP impossibility result
- 23.6.2 The CAP theorem and network partitions
-
23.7 Majority quorums
- 23.7.1 Quorum overlap guarantees
-
23.8 Handling writes during a network partition
-
23.9 Changing cluster membership
-
23.10 Crash faults and Byzantine faults
-
23.11 Comparing Paxos, Raft, Zab and PBFT
-
23.12 Alternatives to application-owned consensus
-
23.13 Building a quorum register in the lab
-
Chapter 24 Consensus algorithms
-
24.1 Raft and Paxos
-
24.2 Raft roles and node state
-
24.3 Terms in Raft
-
24.4 Raft leader election
-
24.5 Raft log replication and commitment
-
24.6 Persisting terms, votes and log entries
- 24.6.1 Crashes during a write
-
24.7 Snapshots and membership changes
-
24.8 Client retries and deduplication
-
24.9 Monitoring and diagnosing a Raft cluster
-
24.10 Testing elections, crashes and partitions
-
24.11 Choosing a coordination service
-
24.12 Building the election lab
-
Chapter 25 Replication and state machines
-
25.1 The replicated state machine
- 25.1.1 Designing replicated commands
-
25.2 Committing, applying and replying to clients
- 25.2.1 Detecting duplicate client requests
-
25.3 Side effects and the outbox pattern
-
25.4 Repairing a follower's log
-
25.5 Snapshots and log compaction
-
25.6 Committed commands that fail to apply
-
25.7 Read freshness and stale-read mitigation
- 25.7.1 Linearizability requirements
-
25.8 Persistence and crash consistency
-
25.9 The cost of replication
-
Chapter 26 Distributed locking
-
26.1 Locking for efficiency and locking for correctness
-
26.2 Atomic command execution in Redis
- 26.2.1 Lost updates with GET and SET
- 26.2.2 Expiring key leases with SET NX PX
- 26.2.3 Checking lock ownership with Lua scripts
- 26.2.4 Limits of atomic commands
-
26.3 Writes after lease expiry
- 26.3.1 Renewing a lease
-
26.4 Fencing tokens and stale-write rejection
-
26.5 Conditional writes in Azure and on-prem storage
-
26.6 Stale-write rejection at the protected resource
-
26.7 Message broker locks
-
26.8 Monitoring and debugging locks
- 26.8.1 Handling acquire, renew and release failures
-
26.9 Choosing a coordination mechanism
-
Part VI State and storage
-
Chapter 27 Stateless services
-
27.1 State in a stateless service
-
27.2 Handling retries across instances
-
27.3 Loading configuration and secrets at startup
- 27.3.1 Managed identities and how long to keep secrets
-
27.4 ASGI applications and worker processes
-
27.5 Liveness and readiness probes
-
27.6 Graceful shutdown
-
27.7 Counters, schedulers and updates across instances
-
27.8 Retrying and limiting load
-
27.9 Validating multi-instance state sharing and connection draining
-
Chapter 28 Distributed data storage
-
28.1 Partitioning keys across nodes
- 28.1.1 Placing replicas in separate failure domains
-
28.2 Choosing a partition key
-
28.3 Rebalancing data while the store is running
-
28.4 Single-leader, multi-leader and leaderless replication
-
28.5 Repairing replicas
- 28.5.1 Deleting data with tombstones
-
28.6 Consistency models
-
28.7 Resolving concurrent writes
-
28.8 Partition limits and hot partitions in Azure
- 28.8.1 Diagnosing a hot partition in Cosmos DB
-
28.9 Partitioning and placement on-prem
-
28.10 Validating ring stability and replica repair
-
Chapter 29 Key-value stores
-
29.1 Redis data structures and commands
- 29.1.1 Python pipeline transactions and optimistic concurrency with WATCH
- 29.1.2 Persistence, eviction, key scans and cluster slots
- 29.1.3 Persistence files and replication
- 29.1.4 Choosing an eviction policy
-
29.2 Refilling a cache and keeping it fresh
-
29.3 Cosmos DB items and consistency levels
- 29.3.1 Request-unit charges and throttling
- 29.3.2 Choosing a partition key and an indexing policy
- 29.3.3 Retries and the change feed
- 29.3.4 Changing a partition key and logical partition size limits
-
29.4 Choosing between a cache and an authoritative store
- 29.4.1 Migrating to Azure Managed Redis
-
29.5 Valkey and PostgreSQL on-prem
-
29.6 Validating key-value storage semantics
-
Part VII Orchestration
-
Chapter 30 Directed acyclic graphs (DAGs)
-
30.1 Validating a graph before it runs
-
30.2 Topological ordering and cycle detection
-
30.3 Running ready nodes on several workers
-
30.4 Deterministic scheduling
-
30.5 Estimating run time with the critical path
-
30.6 Graph levels and redundant edges
-
30.7 Failure policies
-
30.8 Attempts and retries
-
30.9 Storing run state in a database
-
30.10 Resuming a run after a crash
-
30.11 Diagnosing a stuck or failed run
-
30.12 Deriving edges from inputs and outputs
-
Chapter 31 Workflow orchestration
-
31.1 Airflow components and the metadata database
-
31.2 Keeping DAG files fast to import
- 31.2.1 Slow imports and delayed runs
-
31.3 Data intervals and logical dates
-
31.4 Planning capacity for catchup and backfill
-
31.5 Retry-safe and replay-safe tasks
-
31.6 Passing results between tasks with XCom
-
31.7 Sensors and deferrable operators
-
31.8 Concurrency controls
-
31.9 Managing secrets and identities
-
31.10 Deploying and upgrading Airflow
-
31.11 Running Airflow on Azure
-
31.12 Running Airflow on-prem
-
31.13 Alternatives to Airflow
-
Part VIII Systems operations
-
Chapter 32 Containers and Kubernetes
-
32.1 Process isolation options
-
32.2 Linux security controls for containers
-
32.3 From OCI image to running process
-
32.4 Running virtual machines with QEMU and KVM
-
32.5 Managing virtual machines with libvirt
-
32.6 Controlling a running VM with QMP
-
32.7 Application containers and system containers
-
32.8 Kata Containers and the utility VM
-
32.9 The local lab platform
-
32.10 The SigRaft on-prem deployment
-
32.11 Building application container images
-
32.12 Registries, tags and digests
-
32.13 Azure Container Apps and AKS
-
32.14 Kubernetes objects for a stateless service
-
32.15 The Kubernetes Python client
-
32.16 Kubernetes pod scheduling
-
32.17 Workload identity in a container
-
32.18 Rolling updates and the limits of rollback
-
32.19 Moving from a local kind cluster to AKS
-
32.20 Routing requests through Nginx
-
32.21 Health probes, autoscaling and disruption budgets
-
32.22 Namespace isolation and coexisting versions in Kubernetes
- 32.22.1 Kubernetes rolling updates
- 32.22.2 Canary traffic splitting in Kubernetes
- 32.22.3 Blue-green environment switching in Kubernetes
-
Chapter 33 Failure and backpressure
-
33.1 Request deadlines and timeout budget propagation
-
33.2 Exponential backoff and jittered retry policies
- 33.2.1 Backoff, jitter and retry limits
-
33.3 Retry amplification and overload collapse
-
33.4 Circuit breaker states and trip thresholds
-
33.5 Resource pool isolation with bulkheads
-
33.6 Sizing a queue
-
33.7 Admission control and load shedding
-
33.8 Health checks during overload
-
33.9 Detecting and diagnosing service overload
-
33.10 Testing deadlines and overload
-
Chapter 34 Observability
-
34.1 Choosing what telemetry to collect
-
34.2 Structured logging
-
34.3 Request IDs and trace IDs
-
34.4 Metrics and latency distributions
-
34.5 Traces and spans
-
34.6 OpenTelemetry providers, exporters and the collector
- 34.6.1 Baggage and passing context between threads and processes
-
34.7 Azure Monitor and Application Insights
-
34.8 Health checks and service-level health
-
34.9 Service level objectives, error budgets and alerts
-
34.10 Tracing an incident
-
Chapter 35 Operational data analysis
-
35.1 Defining the decision, the population and the observation
-
35.2 Arranging data in tidy tables
-
35.3 NumPy arrays, masks and default settings
-
35.4 Loading and validating data with pandas
-
35.5 Describing a distribution
-
35.6 Confidence intervals and comparing releases
-
35.7 Regression on job duration
-
35.8 Accessible charts and tables
-
35.9 Sampling errors and measurement errors
-
35.10 Effect size, uncertainty and release decisions
-
Part IX The SigRaft on-prem cloud
-
Chapter 36 SigRaft on-prem cloud infrastructure
-
36.1 OpenStack and Kubernetes responsibilities
-
36.2 Validating the configuration and planning changes
-
36.3 Sizing the host and its virtual machines
-
36.4 Separating networks and naming services with DNS
-
36.5 Storage services and backups
-
36.6 Managing identities and secrets
-
36.7 Tracing requests and monitoring the platform
-
36.8 Failure domains, deployment order and teardown
-
36.9 Bridging private infrastructure components to Azure equivalents
-
Chapter 37 On-prem cloud APIs with Python
-
37.1 Adapter contracts bridging private cloud APIs and Azure SDKs
-
37.2 OpenStackSDK service proxies
-
37.3 Kubernetes desired state and rollout readiness
-
37.4 Harbor HTTP clients
-
37.5 Managing rack servers with Redfish
-
37.6 Per-adapter identities
-
37.7 Pagination and resumable reads
-
37.8 Operations, readiness and deadlines
-
37.9 Lost requests and lost responses
-
37.10 Planning and applying resource changes
-
37.11 Client lifetime and logging
-
37.12 Fake clients and tests against live platforms
-
Part X Resource scheduling
-
Chapter 38 Resource-aware job scheduling
-
38.1 Queues, workers and schedulers
-
38.2 Describing a job's resource requirements
-
38.3 Tracking nodes with inventory reports and heartbeats
-
38.4 Rejecting impossible requests and queuing waiting ones
-
38.5 Choosing a node with filtering and scoring
-
38.6 Electing one scheduler and storing reservations
-
38.7 Sending allocations to node queues
-
38.8 Enforcing resource limits on the node
- 38.8.1 Limiting CPU and memory with cgroups
- 38.8.2 Keeping CPUs, GPUs and memory on one NUMA node
- 38.8.3 Limiting which GPUs a job can see
-
38.9 Job states, allocation leases and node failure
-
38.10 Priority, FIFO order and project quotas
-
38.11 Advanced scheduling policies
-
38.12 Comparing Slurm, Kubernetes, Ray and Airflow
-
38.13 Running the scheduler locally, on-prem and on Azure
-
38.14 Logging decisions and securing the scheduler
-
Part XI Release engineering
-
Chapter 39 The completed SigRaft service
-
39.1 Promoting one image by digest
-
39.2 REST and GraphQL interfaces
-
39.3 WebSocket commands and job notifications
- 39.3.1 Authenticating and authorizing WebSocket clients
- 39.3.2 Initial snapshots, event ordering and slow clients
-
39.4 Versioning the service documentation
-
39.5 Reserving resources before dispatching work
-
39.6 Release quality reports
- 39.6.1 Releasing to the on-prem cloud
- 39.6.2 Release reports for Azure
-
39.7 Azure pipelines, identities and approvals
-
39.8 Configuring each environment
- 39.8.1 Reloading configuration without a restart
-
39.9 Multi-strategy deployment pipelines for SigRaft
-
39.10 Traffic draining and zero-downtime service transitions
-
39.11 Configuring hosts with Ansible
-
39.12 Running short remote tasks with Fabric
-
39.13 Validating a deployment with telemetry
- 39.13.1 Read-only hardware inventory
-
39.14 Diagnosing and recovering from a failed rollout
Appendices
-
Appendix A The lab toolchain
-
A.1 Installing and pinning the toolchain
-
A.2 Inspecting lab code at the Python prompt
-
A.3 Commands for creating projects and running gates
-
A.4 Exit statuses
-
A.5 Configuring a lab project
-
A.6 Configuration precedence
-
A.7 The tools behind each gate
-
A.8 Skipped gates
-
A.9 Publishing results as JUnit XML
-
A.10 Running the gates offline
-
A.11 Running every lab's quality gates
-
Appendix B Linux command-line tutorial and reference
-
B.1 The shell, terminal and working directory
-
B.2 Creating, copying, moving and removing files
-
B.3 Reading and writing text
-
B.4 Redirection and pipelines
-
B.5 Finding files and matching text
-
B.6 Command history
-
B.7 Environment variables, processes and jobs
-
B.8 Connecting with SSH
-
B.9 A first Bash script
-
B.10 Editing files with Vim
-
B.11 Diagnostic commands
- B.11.1 Sockets, connections and processes
- B.11.2 Name resolution and IP routing
- B.11.3 Firewall rules and packet capture
- B.11.4 Tracing system calls and Python stacks
- B.11.5 Cgroup limits and out-of-memory kills
- B.11.6 Inspecting a systemd service
- B.11.7 Simulating network delay, loss and reordering
- B.11.8 Signals and graceful shutdown
- B.11.9 Kernel network settings
- B.11.10 Permissions for diagnostic commands
- B.11.11 Cleaning up after diagnosis
-
Appendix C Git development flow tutorial
-
C.1 Configuring Git and getting help
-
C.2 The working tree, staging area and first commit
-
C.3 Ignoring generated and private files
-
C.4 Reading history and comparing revisions
-
C.5 Branches and HEAD
-
C.6 Working with remote repositories
-
C.7 Choosing a branching method
-
C.8 Merging branches
-
C.9 Resolving merge conflicts
-
C.10 Rebasing a private branch
-
C.11 Cherry-picking commits
-
C.12 Creating pull requests from the command line
-
C.13 Submodules
-
C.14 Undoing changes and checking repository status
References