Getting good visibility into cloud-native applications requires a lot of work. In microservices architectures, the payload (lines of code, amount of functionality) of each service is reduced significantly. This increases the amount of individual services that make up a given application.

Managing this large volume in terms of performance and security monitoring is hard. How do you know which containers are running, what dependencies each has and what the health of the individual containers, but also the health of the overarching application is?

This is where Sysdig comes in. Sysdig is a performance and security monitoring platform for cloud native applications.

Company roots

Sysdig is founded by Loris Degioanni. He’s the co-creator of the network packet analyzer WireShark, and has applied many of basic principles of that work to Sysdig.

Loris Degioanni

Sysdig set out to solve packet-level telemetry without access to the underlying network stack to actually capture packets. Cloud don’t generally give you this kind of access to their network stack, and with good reason. Instead, they use open source, Linux-kernel instrumentation via eBPF.

eBPF in-kernel instrumentation

Instead of instrumenting each container individually or adding sidecars (doubling the amount of running containers), Sysdig uses kernel-native instrumentation via eBPF to capture system calls and other OS events from containers. This requires zero changes to how containers are built or run.

With every host under management (in all your clusters) instrumented, you get to see all containers’ activities. All Containers. All their interactions. From a security perspective, this is brilliant. Even if there’s a rogue container running for only seconds at a time, Sysdig will catch it. Much like Wireshark, Sysdig uses network data to map dependencies across containers and see what traffic is deemed ‘normal’ or anomalous.

All this data is pushed into the Sysdig analytics engine for later forensic analysis, performance troubleshooting and more.

sysdig agent ebpf

Sysdig leverages the increasingly popular eBPF in-kernel virtual machine. This allows trusted and verified programs, like Sysdig’s monitoring and tracing code, to run in the kernel securely. eBPF lets Sysdig filter and monitor network traffic, system calls and file system activity in a performant, non-intrusive way. The eBPF programs forward the captures to the Sysdig Agent, which itself runs on a container on each host. The container does additional processing, and sends it up to the Sysdig Platform for correlation across hosts. The Agent container is available via the various Operator hubs for easy deployment across your environment.

Note how this is different from other instrumentation approaches that need instrumentation at the application level, but makes it easy to include for open source package maintainers. The difference here is that projects like OpenTracing make it easier to package up standard instrumentation with a package for easy distribution, while eBPF is dynamically inserted during runtime. There’s some down-sides to this approach, too. Managed container services (like AKS, EKS, GKE) may not offer access to their container hosts to Sysdig’s eBPF programs.

eBPF in itself is not an easy thing to complete grasp. If you want to learn more, I recommend you spend some time on Sysdig’s blog:

  1. Sysdig and Falco now powered by eBPF.
  2. Introducing container observability with eBPF + Sysdig
  3. The art of writing eBPF programs: a primer

Open Core

Sysdig has an open core approach to its offerings. At the most basic level, all functionality comes from that kernel-level instrumentation.

The sysdig tool is the core tool and gives command-line observability for containers. It’s a simple way to get deep system visibility of container behaviour and leans on Sysdig’s in-kernel approach for capturing telemetry. This tool is meant for single containers and single hosts.

Inspect is a open source graphical interface for troubleshooting and security forensics that ingests sysdig’s data collection.

Falco is a rules-based detection engine for runtime security to detect behavioural anomalies and can take policy-based action on malicious activity. Containers are usually static in their behaviour. The expected behaviour is known, due to the configuration-as-code characteristics of containers, that pre-defined how a container should behave as part of creating the container image specification. Falco watches for deviations and generates alerts when something suspicious pops up. It also lets you create playbooks that pre-defines the action the systems will take if a deviation is observed.

Sysdig also builds on other open source tools, like Anchore (container image vulnerability scanning) and Prometheus (monitoring of Kubernetes) to create a wider reach of their platform without re-inventing the wheel.

Sysdig Platform

We see that Sysdig gets packet-level telemetry for running containers from eBPF and the Sysdig Agent container. But the open source sysdig and Inspect tools are not suitable for the scale that enterprises operate on; they need a multi-cloud and multi-cluster solution for monitoring.

Sysdig Platform integrated all of the open source tools into a single, unified view of container health. Integrating performance and security monitoring in a single platform solves a problem many organizations face: communication is hard. Most teams will have a narrow, specific responsibility, like security, deployment or development. They’ll all have a different definition of healthy (security versus performance, for instance). And they’ll have a hard time communicating their narrow perspective to other teams.

Working from a single platform creates a common, shared understanding of what health means (and how multi-facetted it, in reality, is). It shows performance metrics, compliance checks and security events across the infrastructure to many teams, in the same way. This helps teams highlight the events and containers that have performance or security and compliance issues that need immediate attention.

This overview then provides deep contextual information that allows users to dive deep into Sysdig Secure and Sysdig Monitor to analyze performance metrics, compliance dashboards, security forensics, and more to isolate and remediate problems faster than ever before.

Sysdig build, run, respond.

Sysdig Monitor provides the run-time monitoring, dashboarding, alerting and trace-driven troubleshooting. Monitor uses the open source sysdig tool.

Sysdig Secure is the security part of the equation with vulnerability management, compliance (250+ out-of-the-box compliance checks), runtime security, anomaly detection and forensics for post-mortem improvements. Sysdig Secure uses the open source Falco, Inspect and Anchore tools.

Sysdig is not just for production. In addition, it helps developers by trying to catch faults (like known security vulnerabilities) before they even make it to production.

Diggin’ it!

Sysdig marries the old and proven with the new and innovative. What makes Sysdig, well, dig, is the age-old idea of capturing network packets as a monitoring and inspection method, but with a new, cloud-native, look. Leveraging eBPF and the Sysdig Agent container, Sysdig knows all there is to know about containers.

They use this vast amount of data for performance monitoring, security forensics, vulnerability scanning, behavioural anomaly detection, application tracing and run-time compliance checks.

This is ground-breaking technology.

ground-breaking digging

Being able to do application tracing across all container without modifying them to inject instrumentation is a hard problem to solve.
Collecting that much data about ephemeral workloads is no small feat, and is a life-saver when troubleshooting for performance issues.
Being able to to forensics and anomaly detection is great for complex microservices landscapes where no one team controls it all but security teams still need to be compliant.

I’m genuinely impressed by the elegance of Sysdig’s instrumentation approach, and what they can do with all that data they collect.