Cloud Native DevOps with Kubernetes

Foreword by NGINX. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xix

Foreword by Ihor Dvoretskyi. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxi

Preface. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxiii

1. Revolution in the Cloud. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1

The Creation of the Cloud 2

Buying Time 3

Infrastructure as a Service 3

The Dawn of DevOps 3

Nobody Understands DevOps 5

The Business Advantage 5

Infrastructure as Code 6

Learning Together 6

The Coming of Containers 7

The State of the Art 7

Thinking Inside the Box 8

Putting Software in Containers 8

Plug and Play Applications 9

Conducting the Container Orchestra 10

Kubernetes 11

From Borg to Kubernetes 11

What Makes Kubernetes So Valuable? 11

Will Kubernetes Disappear? 13

Kubernetes Doesn’t Do It All 13

Cloud Native 14

The Future of Operations 16

Distributed DevOps 17

Some Things Will Remain Centralized 17

Developer Productivity Engineering 17

You Are the Future 18

Summary 19

2. First Steps with Kubernetes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

Running Your First Container 21

Installing Docker Desktop 22

What Is Docker? 22

Running a Container Image 22

The Demo Application 23

Looking at the Source Code 23

Introducing Go 24

How the Demo App Works 24

Building a Container 25

Understanding Dockerfiles 25

Minimal Container Images 26

Running docker image build 26

Naming Your Images 27

Port Forwarding 27

Container Registries 28

Authenticating to the Registry 28

Naming and Pushing Your Image 28

Running Your Image 29

Hello, Kubernetes 29

Running the Demo App 30

If the Container Doesn’t Start 30

Minikube 31

Summary 31

3. Getting Kubernetes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33

Cluster Architecture 33

The Control Plane 34

Node Components 35

High Availability 35

The Costs of Self-Hosting Kubernetes 37

It’s More Work Than You Think 37

It’s Not Just About the Initial Setup 38

Tools Don’t Do All the Work for You 39

Kubernetes Is Hard 39

Administration Overhead 39

Start with Managed Services 40

Managed Kubernetes Services 41

Google Kubernetes Engine (GKE) 41

Cluster Autoscaling 42

Amazon Elastic Container Service for Kubernetes (EKS) 42

Azure Kubernetes Service (AKS) 43

OpenShift 43

IBM Cloud Kubernetes Service 43

Heptio Kubernetes Subscription (HKS) 43

Turnkey Kubernetes Solutions 44

Stackpoint 44

Containership Kubernetes Engine (CKE) 44

Kubernetes Installers 44

kops 45

Kubespray 45

TK8 45

Kubernetes The Hard Way 45

kubeadm 46

Tarmak 46

Rancher Kubernetes Engine (RKE) 46

Puppet Kubernetes Module 46

Kubeformation 46

Buy or Build: Our Recommendations 47

Run Less Software 47

Use Managed Kubernetes if You Can 48

But What About Vendor Lock-in? 48

Use Standard Kubernetes Self-Hosting Tools if You Must 49

When Your Choices Are Limited 49

Bare-Metal and On-Prem 49

Clusterless Container Services 50

Amazon Fargate 50

Azure Container Instances (ACI) 51

Summary 51

4. Working with Kubernetes Objects. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53

Deployments 53

Supervising and Scheduling 54

Restarting Containers 54

Querying Deployments 55

Pods 55

ReplicaSets 56

Maintaining Desired State 57

The Kubernetes Scheduler 58

Resource Manifests in YAML Format 59

Resources Are Data 59

Deployment Manifests 59

Using kubectl apply 60

Service Resources 60

Querying the Cluster with kubectl 63

Taking Resources to the Next Level 64

Helm: A Kubernetes Package Manager 64

Installing Helm 65

Installing a Helm Chart 65

Charts, Repositories, and Releases 66

Listing Helm Releases 67

Summary 67

5. Managing Resources. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69

Understanding Resources 69

Resource Units 70

Resource Requests 70

Resource Limits 71

Keep Your Containers Small 72

Managing the Container Life Cycle 72

Liveness Probes 72

Probe Delay and Frequency 73

Other Types of Probes 73

gRPC Probes 74

Readiness Probes 74

File-Based Readiness Probes 75

minReadySeconds 75

Pod Disruption Budgets 76

Using Namespaces 77

Working with Namespaces 78

What Namespaces Should I Use? 78

Service Addresses 79

Resource Quotas 79

Default Resource Requests and Limits 80

Optimizing Cluster Costs 81

Optimizing Deployments 82

Optimizing Pods 83

Vertical Pod Autoscaler 84

Optimizing Nodes 84

Optimizing Storage 85

Cleaning Up Unused Resources 86

Checking Spare Capacity 88

Using Reserved Instances 88

Using Preemptible (Spot) Instances 89

Keeping Your Workloads Balanced 91

Summary 92

6. Operating Clusters. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95

Cluster Sizing and Scaling 95

Capacity Planning 96

Nodes and Instances 98

Scaling the Cluster 101

Conformance Checking 102

CNCF Certification 103

Conformance Testing with Sonobuoy 104

Validation and Auditing 105

K8Guard 106

Copper 106

kube-bench 107

Kubernetes Audit Logging 107

Chaos Testing 107

Only Production Is Production 108

chaoskube 108

kube-monkey 109

PowerfulSeal 109

Summary 110

7. Kubernetes Power Tools. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111

Mastering kubectl 111

Shell Aliases 111

Using Short Flags 112

Abbreviating Resource Types 112

Auto-Completing kubectl Commands 113

Getting Help 113

Getting Help on Kubernetes Resources 114

Showing More Detailed Output 114

Working with JSON Data and jq 114

Watching Objects 115

Describing Objects 116

Working with Resources 116

Imperative kubectl Commands 116

When Not to Use Imperative Commands 117

Generating Resource Manifests 118

Exporting Resources 118

Diffing Resources 119

Working with Containers 119

Viewing a Container’s Logs 119

Attaching to a Container 121

Watching Kubernetes Resources with kubespy 121

Forwarding a Container Port 121

Executing Commands on Containers 122

Running Containers for Troubleshooting 122

Using BusyBox Commands 124

Adding BusyBox to Your Containers 124

Installing Programs on a Container 125

Live Debugging with kubesquash 125

Contexts and Namespaces 126

kubectx and kubens 128

kube-ps1 128

Kubernetes Shells and Tools 129

kube-shell 129

Click 129

kubed-sh 130

Stern 130

Building Your Own Kubernetes Tools 130

Summary 131

8. Running Containers. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133

Containers and Pods 133

What Is a Container? 134

What Belongs in a Container? 135

What Belongs in a Pod? 136

Container Manifests 136

Image Identifiers 137

The latest Tag 138

Container Digests 138

Base Image Tags 139

Ports 139

Resource Requests and Limits 139

Image Pull Policy 140

Environment Variables 140

Container Security 141

Running Containers as a Non-Root User 142

Blocking Root Containers 142

Setting a Read-Only Filesystem 143

Disabling Privilege Escalation 143

Capabilities 144

Pod Security Contexts 145

Pod Security Policies 145

Pod Service Accounts 146

Volumes 147

emptyDir Volumes 147

Persistent Volumes 148

Restart Policies 149

Image Pull Secrets 149

Summary 150

9. Managing Pods. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153

Labels 153

What Are Labels? 153

Selectors 154

More Advanced Selectors 155

Other Uses for Labels 156

Labels and Annotations 156

Node Affinities 157

Hard Affinities 158

Soft Affinities 158

Pod Affinities and Anti-Affinities 159

Keeping Pods Together 159

Keeping Pods Apart 160

Soft Anti-Affinities 161

When to Use Pod Affinities 161

Taints and Tolerations 162

Pod Controllers 163

DaemonSets 164

StatefulSets 165

Jobs 166

Cronjobs 167

Horizontal Pod Autoscalers 167

PodPresets 169

Operators and Custom Resource Definitions (CRDs) 170

Ingress Resources 171

Ingress Rules 172

Terminating TLS with Ingress 172

Ingress Controllers 173

Istio 174

Envoy 175

Summary 175

10. Configuration and Secrets. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 179

ConfigMaps 179

Creating ConfigMaps 180

Setting Environment Variables from ConfigMaps 181

Setting the Whole Environment from a ConfigMap 183

Using Environment Variables in Command Arguments 184

Creating Config Files from ConfigMaps 184

Updating Pods on a Config Change 186

Kubernetes Secrets 187

Using Secrets as Environment Variables 187

Writing Secrets to Files 188

Reading Secrets 189

Access to Secrets 190

Encryption at Rest 190

Keeping Secrets 191

Secrets Management Strategies 191

Encrypt Secrets in Version Control 191

Store Secrets Remotely 192

Use a Dedicated Secrets Management Tool 193

Recommendations 193

Encrypting Secrets with Sops 194

Introducing Sops 194

Encrypting a File with Sops 195

Using a KMS Backend 196

Summary 197

11. Security and Backups. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 199

Access Control and Permissions 199

Managing Access by Cluster 199

Introducing Role-Based Access Control (RBAC) 200

Understanding Roles 201

Binding Roles to Users 201

What Roles Do I Need? 202

Guard Access to Cluster-Admin 202

Applications and Deployment 203

RBAC Troubleshooting 204

Security Scanning 204

Clair 204

Aqua 205

Anchore Engine 205

Backups 206

Do I Need to Back Up Kubernetes? 206

Backing Up etcd 206

Backing Up Resource State 207

Backing Up Cluster State 207

Large and Small Disasters 208

Velero 208

Monitoring Cluster Status 211

kubectl 211

CPU and Memory Utilization 213

Cloud Provider Console 213

Kubernetes Dashboard 214

Weave Scope 216

kube-ops-view 216

node-problem-detector 216

Further Reading 217

Summary 217

12. Deploying Kubernetes Applications. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 219

Building Manifests with Helm 219

What’s Inside a Helm Chart? 220

Helm Templates 221

Interpolating Variables 222

Quoting Values in Templates 223

Specifying Dependencies 223

Deploying Helm Charts 223

Setting Variables 223

Specifying Values in a Helm Release 224

Updating an App with Helm 225

Rolling Back to Previous Versions 225

Creating a Helm Chart Repo 226

Managing Helm Chart Secrets with Sops 226

Managing Multiple Charts with Helmfile 228

What’s in a Helmfile? 228

Chart Metadata 229

Applying the Helmfile 230

Advanced Manifest Management Tools 231

ksonnet 231

Kapitan 232

kustomize 233

kompose 233

Ansible 234

kubeval 234

Summary 235

13. Development Workflow. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 237

Development Tools 237

Skaffold 237

Draft 238

Telepresence 238

Knative 238

Deployment Strategies 239

Rolling Updates 239

Recreate 240

maxSurge and maxUnavailable 240

Blue/Green Deployments 241

Rainbow Deployments 242

Canary Deployments 242

Handling Migrations with Helm 243

Helm Hooks 243

Handling Failed Hooks 244

Other Hooks 244

Chaining Hooks 244

Summary 245

14. Continuous Deployment in Kubernetes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 247

What Is Continuous Deployment? 247

Which CD Tool Should I Use? 248

Jenkins 248

Drone 249

Google Cloud Build 249

Concourse 249

Spinnaker 249

GitLab CI 250

Codefresh 250

Azure Pipelines 250

CD Components 250

Docker Hub 250

Gitkube 250

Flux 251

Keel 251

A CD Pipeline with Cloud Build 251

Setting Up Google Cloud and GKE 251

Forking the Demo Repository 252

Introducing Cloud Build 252

Building the Test Container 252

Running the Tests 253

Building the Application Container 253

Validating the Kubernetes Manifests 254

Publishing the Image 254

Git SHA Tags 254

Creating the First Build Trigger 255

Testing the Trigger 255

Deploying from a CD Pipeline 257

Creating a Deploy Trigger 259

Optimizing Your Build Pipeline 259

Adapting the Example Pipeline 260

Summary 260

15. Observability and Monitoring. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 263

What Is Observability? 263

What Is Monitoring? 263

Black-Box Monitoring 263

What Does “Up” Mean? 265

Logging 266

Introducing Metrics 268

Tracing 269

Observability 270

The Observability Pipeline 271

Monitoring in Kubernetes 272

External Black-Box Checks 272

Internal Health Checks 274

Summary 276

16. Metrics in Kubernetes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 277

What Are Metrics, Really? 277

Time Series Data 278

Counters and Gauges 278

What Can Metrics Tell Us? 279

Choosing Good Metrics 279

Services: The RED Pattern 280

Resources: The USE Pattern 281

Business Metrics 282

Kubernetes Metrics 283

Analyzing Metrics 286

What’s Wrong with a Simple Average? 287

Means, Medians, and Outliers 287

Discovering Percentiles 288

Applying Percentiles to Metrics Data 288

We Usually Want to Know the Worst 290

Beyond Percentiles 290

Graphing Metrics with Dashboards 291

Use a Standard Layout for All Services 291

Build an Information Radiator with Master Dashboards 292

Dashboard Things That Break 293

Alerting on Metrics 295

What’s Wrong with Alerts? 295

On-call Should Not Be Hell 296

Urgent, Important, and Actionable Alerts 296

Track Your Alerts, Out-of-Hours Pages, and Wake-ups 297

Metrics Tools and Services 298

Prometheus 298

Google Stackdriver 300

AWS Cloudwatch 301

Azure Monitor 301

Datadog 301

New Relic 302

Summary 303

Afterword. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 305

Index. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 307

......(更多)