Ycsecurity/auditlogs/export-auditlogs-to-ELK_k8s

Yandex.Cloud: Analyzing K8s security logs in ELK: audit logs, Policy Engine, Falco

Version

Version-2.0

• Changelog:
  • Changed the method of deployment. Deprecation of virtual machines as a worker engine to deployments in k8s. Thanks to "Hilbert Team" for contribution

Docker images:

• cr.yandex/sol/k8s-events-siem-worker:2.0.0.

Version-2.0

• Changelog:
  • Added support for automatic Kyverno installation with policies in the audit mode.
• Docker images:
  • cr.yandex/sol/k8s-events-siem-worker:1.1.0.

Table of contents

• Description
• Link to the solution "Collecting, monitoring, and analyzing audit logs in Yandex Managed Service for Elasticsearch (ELK)"")
• Generic diagram
• Terraform description
• Content update process
• Optional manual actions

Description

Here are the out-of-the-box features of the solution: ☑️ Collect K8s audit logs in Managed ELK SIEM.

• ☑️ Install Falco and collect its Alerts in Managed ELK SIEM.
• ☑️ Install Kyverno with the Pod Security Standards (Restricted) policies in the audit mode and collect its Alerts (Policy Reports) using Policy Reporter.
• ☑️ Import Security Content: dashboards, detection rules, and so on (see the Security Content section) in Managed ELK SIEM to enable analysis and response to information security events.
• ☑️ This also includes importing Security Content for OPA Gatekeeper (in the enforce mode). You can install OPA Gatekeeper manually if needed.
• ☑️ Create indexes in two replicas, set up the basic rollover policy (creating of new indexes every thirty days or when 50 GB are reached) to enable provisioning of high data availability and to set up data snapshots in S3, see recommendations.

Link to the solution "Collecting, monitoring, and analyzing audit logs in Yandex Managed Service for Elasticsearch (ELK)"

The solution "Collecting, monitoring, and analyzing audit logs in Yandex Managed Service for Elasticsearch (ELK)" contains information on how to install Yandex Managed Service for Elasticsearch (ELK) and collect logs from Audit Trails in it.

Generic diagram

Description of imported ELK (Security Content) object

See a detailed description of the objects here.

Terraform description

The solution consist of terraform module:

• It accepts the following input:
  • folder_id: The ID of the folder where the cluster is hosted.
  • cloud_id: The ID of the cloud where the cluster is hosted.
  • cluster_name: The name of the Kubernetes cluster.
  • elastic_server: The FQDN address of the ELK installation
  • elastic_pw and elastic_user: The ELK user credentials for event import
  • service_account_id: The ID of the service account that can write to the bucket and has the ymq.admin role.
  • log_bucket_name: The name of the bucket that will create module to save logs to.
  • auditlog_enabled: true or false (enables/disables sending of K8s audit logs to ELK).
  • falco_enabled: true or false (enables/disables sending of Falco alerts to ELK).
  • kyverno_enabled: true or false — (enables/disables sending of Kyverno alerts to ELK).
• Functionality:
  • Create a static key for the service account.
  • Create a function and a trigger for writing cluster logs to S3.
  • Install Falco and pre-configured falcosidekick that will send logs to S3.
  • Install Kyverno and pre-configured Policy Reporter that will send logs to S3.
  • Create YMQ queues with log file names in S3.
  • Create functions to push file names from S3 to YMQ.
  • Create triggers for interaction between queues and functions.
  • Create deployments in k8s with worker containers that import events from S3 to ELK.

Prerequisites:

• :white_check_mark: Cluster Managed K8s.
• :white_check_mark: Managed ELK.
• :white_check_mark: A service account that can write to the bucket and has the ymq.admin role.

Example of calling modules:

See the example of calling modules in /examples/README.md

Content update process

We recommend subscribing to this repository to receive update notifications.

For content updates, make sure that you are running the latest available image version: cr.yandex/sol/k8s-events-siem-worker:latest

You can update the container as follows: You can re-create the deployments in k8s via Terraform (change worker_docker_image env in tfvars and run terraform apply).

Optional manual actions

Installing OPA Gatekeeper (Helm)

If you prefer OPA Gatekeeper to Kyverno, set the value kyverno_enabled to false when calling the module, then run the manual installation:

• Install OPA Gatekeeper using Helm.
• Select and install the required constraint template and constraint from gatekeeper-library.
• Installation example.

Recommendations for setting up retention, rollover, and snapshots:

Recommendations for setting up retention, rollover, and snapshots