The rise of containerized infrastructure has caused us to rethink the way we build and deploy our applications. But, with all that speed and flexibility comes challenges, especially when it comes to maintaining visibility into your (often multi-generational) infrastructure.<\/p>\n
In this post, I\u2019ll discuss some of the challenges to monitoring containers and Kubernetes<\/a> (the go-to for container orchestration), talk about some of the data sources and a few methods for monitoring Kubernetes, and walk you through a tutorial on monitoring containers with Sensu<\/a>, a multi-cloud monitoring tool.<\/p>\n In a containerized world, your applications are constantly moving, making it even more difficult to keep tabs on them. It\u2019s also the norm to have distributed applications, which causes a whole new set of problems when it comes to observability. Said another way, this new reality of distributed, constantly moving infrastructure means there are many smaller pieces to monitor. Similarly, it\u2019s important to keep track of the labels and annotations associated with pods and containers.<\/p>\n When it comes to monitoring Kubernetes<\/a>, there are essentially four data sources you\u2019re plugging into your monitoring tool:<\/p>\n Of these data sources, we tend to prefer kube-state-metrics, as it gives you most of what you need without overloading you with information. I\u2019ll use that for my tutorial of container monitoring with Sensu, but wanted to note that Sensu can also scrape Prometheus metrics (but that\u2019s a whole other post \u2014 for now, feel free to check out our Prometheus collector asset<\/a>).<\/p>\n Using the sidecar pattern, you can deploy a Sensu agent alongside your application container. A sidecar approach makes it possible for each Kubernetes pod to host your application container alongside other containers running support processes, such as the Sensu agent. Since all containers running inside a pod all share the same network space, your applications can talk to Sensu as if they were running in the same container.<\/p>\n In this tutorial, we\u2019ll install and configure sensuctl (Sensu\u2019s command-line tool), deploy the Sensu backend using a Kubernetes deployment, and deploy Sensu agent sidecars. This tutorial assumes you have Kubernetes installed. If you don\u2019t, take a look at setting up Minikube<\/a> which will make things easier for the tutorial.<\/p>\n Download the sensu-kube-demo repo<\/a>. Use this sensu-backend.yaml file<\/a> from the repo and the following command to deploy Sensu:<\/p>\n The You will also want to establish a Kubernetes ingress policy and host DNS configuration that will allow you to communicate with the sensu-backend running inside of the Kubernetes node. For the rest of the tutorial, I will assume you have configured things such that sensu.local tcp port 80 maps to the sensu-backend pod\u2019s TCP port 8080, the default port for the sensu-backend API service. The sensu-kube-demo repository<\/a> includes an appropriate nginx ingress policy example.<\/p>\n sensuctl is a command-line tool for managing resources within Sensu. It works by calling Sensu\u2019s HTTP API to create, read, update, and delete resources, events, and entities. The below instructions are for Ubuntu \u2014 check out our documentation for sensuctl installation on CentOS, Windows, and macOS.<\/p>\n To install sensuctl on Ubuntu:<\/p>\n In order to use sensuctl from your workstation to communicate to the containerized sensu-backend, you\u2019ll need to make sure to configure your Kubernetes ingress controller and network DNS settings to allow access to the sensu-backend API outside of the Kubernetes cluster.<\/p>\n To save time, for this tutorial you can alternatively use the sensuctl provided in the sensu-backend container. In general you\u2019ll want to use sensuctl installed natively in your workstation environment for ease of operation.<\/p>\n If you are using the sensuctl configured on your workstation, enter the following to configure sensuctl:<\/p>\n Please note you\u2019ll need to adjust the URL to match your Kubernetes cluster ingest configuration and DNS configuration.<\/p>\n Open up a new terminal window and start a shell inside the sensu-backend pod:<\/p>\n You can discover the correct pod name with:<\/p>\n Here you can use the localhost URL as if the sensu-backend were running locally, because your sensuctl is operating in the same pod, sharing the pod network with the sensu-backend container.<\/p>\n All the agents used in this tutorial operate in a Sensu namespace named demo. We\u2019ll need to create the namespace now using sensuctl:<\/p>\n We can optionally reconfigure sensuctl to use the demo namespace as default:<\/p>\n Using the aforementioned sidecar pattern and the dummy.sensu.yaml file<\/a>, enter the following to deploy Sensu agent sidecars for two example app instances using a Kubernetes deployment:<\/p>\n If you have set the default namespace to demo, use sensuctl entity list to see the agent containers, otherwise, use sensuctl entity list –namespace demo to set the namespace explicitly. Each agent is running in a separate pod as a sidecar to support the dummy application as per the deployment configuration in the dummy.sensu.yaml file. From here you can use sensuctl to create Sensu checks to examine the running dummy service in each pod by using the dummy subscription, or a specific pod using the specific agent associated with each agent.<\/p>\n For more on container monitoring with Sensu \u2014 including using the Sensu web UI to view events and setting up workflows to Slack and InfluxDB \u2014 check out our interactive tutorial<\/a>. Thanks for reading, and happy monitoring!<\/p>\nContainer monitoring: the challenges<\/h2>\n
Monitoring Kubernetes: the data sources<\/h2>\n
\n
Monitoring Kubernetes: the sidecar pattern<\/h2>\n
Tutorial<\/h2>\n
1. Deploy the Sensu backend using a Kubernetes deployment<\/h3>\n
kubectl create -f go\/deploy\/sensu-backend.yaml<\/code><\/pre>\ngo\/deploy\/<\/code> path reference is relative to the top directory of the repository checkout.<\/p>\n2. [Optional] Install sensuctl on your workstation<\/h3>\n
\n
curl -s https:\/\/packagecloud.io\/install\/repositories\/sensu\/stable\/script.deb.sh | sudo bash<\/code><\/pre>\n\n
sudo apt-get install sensu-go-cli<\/code><\/pre>\n3a. Configure sensuctl to use the built-in admin user on your workstation<\/h3>\n
sensuctl configure -n \\\n--username 'admin'\\\n--password 'P@assw0rd!'\\\n--namespace default\\\n--url 'http:\/\/sensu.local' <\/code><\/pre>\n3b. Configure sensuctl to use the built-in admin user inside the cluster<\/h3>\n
kubectl exec -it sensu-backend-<replace> -- \/bin\/sh<\/code><\/pre>\nkubectl get pods -l app=sensu-backend<\/code><\/pre>\nsensuctl configure -n \\\n--username 'admin'\\\n--password 'P@assw0rd!'\\\n--namespace default\\\n--url 'http:\/\/localhost:8080' <\/code><\/pre>\n4. Create the Sensu namespace for the dummy application<\/h3>\n
sensuctl namespace create demo<\/code><\/pre>\nsensuctl configure -n \\\n--username 'admin'\\\n--password 'P@assw0rd!'\\\n--namespace demo\\<\/code><\/pre>\n5. Deploy Sensu agent sidecars<\/h3>\n
kubectl apply -f go\/deploy\/dummy.sensu.yaml<\/code><\/pre>\n
\n