deploy/loki-distributed/README.md.gotmpl

{{ template "chart.header" . }}

{{ template "chart.versionBadge" . }}{{ template "chart.typeBadge" . }}{{ template "chart.appVersionBadge" . }}

{{ template "chart.description" . }}

{{ template "chart.sourcesSection" . }}

{{ template "chart.requirementsSection" . }}

## Chart Repo

Add the following repo to use the chart:

```console
helm repo add grafana https://grafana.github.io/helm-charts
```

## Upgrading

### Upgrading an existing Release to a new major version

Major version upgrades listed here indicate that there is an incompatible breaking change needing manual actions.

### From 0.68.x to 0.69.0
The in-memory `fifocache` has been renamed to more general `embedded_cache`, which currently doesn't have a `max_size_items` attribute.
```yaml
loki:
  config: |
    chunk_store_config:
      chunk_cache_config:
        embedded_cache:
          enabled: false 
```

`compactor_address` has to be explicitly set in the `common` section of the config.
```yaml
loki:
  config: |
    common:
      compactor_address: {{"{{"}} include "loki.compactorFullname" . {{"}}"}}:3100
```

### From 0.41.x to 0.42.0
All containers were previously named "loki". This version changes the container names to make the chart compatible with the loki-mixin. Now the container names correctly reflect the component (querier, distributor, ingester, ...). If you are using custom prometheus rules that use the container name you probably have to change them.

### From 0.34.x to 0.35.0
This version updates the `Ingress` API Version of the Loki Gateway component to `networking.k8s.io/v1` of course given that the cluster supports it. Here it's important to notice the change in the `values.yml` with regards to the ingress configuration section and its new structure.
```yaml
gateway:
  ingress:
    enabled: true
    # Newly added optional property
    ingressClassName: nginx
    hosts:
      - host: gateway.loki.example.com
        paths:
          # New data structure introduced
          - path: /
            # Newly added optional property
            pathType: Prefix
```

### From 0.30.x to 0.31.0
This version updates the `podManagementPolicy` of running the Loki components as `StatefulSet`'s to `Parallel` instead of the default `OrderedReady` in order to allow better scalability for Loki e.g. in case the pods weren't terminated gracefully. This change requires a manual action deleting the existing StatefulSets before upgrading with Helm.
```bash
# Delete the Ingesters StatefulSets
kubectl delete statefulset RELEASE_NAME-loki-distributed-ingester -n LOKI_NAMESPACE --cascade=orphan
# Delete the Queriers StatefulSets
kubectl delete statefulset RELEASE_NAME-loki-distributed-querier -n LOKI_NAMESPACE --cascade=orphan
```

{{ template "chart.valuesSection" . }}

## Components

The chart supports the components shown in the following table.
Ingester, distributor, querier, and query-frontend are always installed.
The other components are optional.

| Component | Optional | Enabled by default |
| --- | --- | --- |
| gateway |  ✅ |  ✅ |
| ingester |  ❎ | n/a |
| distributor |  ❎ | n/a |
| querier |  ❎ | n/a |
| query-frontend |  ❎ | n/a |
| table-manager |  ✅ |  ❎ |
| compactor |  ✅ |  ❎ |
| ruler |  ✅ |  ❎ |
| index-gateway |  ✅ |  ❎ |
| memcached-chunks |  ✅ |  ❎ |
| memcached-frontend |  ✅ |  ❎ |
| memcached-index-queries |  ✅ |  ❎ |
| memcached-index-writes |  ✅ |  ❎ |

## Configuration

This chart configures Loki in microservices mode.
It has been tested to work with [boltdb-shipper](https://grafana.com/docs/loki/latest/operations/storage/boltdb-shipper/)
and [memberlist](https://grafana.com/docs/loki/latest/configuration/#memberlist_config) while other storage and discovery options should work as well.
However, the chart does not support setting up Consul or Etcd for discovery,
and it is not intended to support these going forward.
They would have to be set up separately.
Instead, memberlist can be used which does not require a separate key/value store.
The chart creates a headless service for the memberlist which ingester, distributor, querier, and ruler are part of.

----

**NOTE:**
In its default configuration, the chart uses `boltdb-shipper` and `filesystem` as storage.
The reason for this is that the chart can be validated and installed in a CI pipeline.
However, this setup is not fully functional.
Querying will not be possible (or limited to the ingesters' in-memory caches) because that would otherwise require shared storage between ingesters and queriers
which the chart does not support and would require a volume that supports `ReadWriteMany` access mode anyways.
The recommendation is to use object storage, such as S3, GCS, MinIO, etc., or one of the other options documented at https://grafana.com/docs/loki/latest/storage/.

Alternatively, in order to quickly test Loki using the filestore, the [single binary chart](https://github.com/grafana/helm-charts/tree/main/charts/loki) can be used.

----

### Directory and File Locations

* Volumes are mounted to `/var/loki`. The various directories Loki needs should be configured as subdirectories (e. g. `/var/loki/index`, `/var/loki/cache`). Loki will create the directories automatically.
* The config file is mounted to `/etc/loki/config/config.yaml` and passed as CLI arg.

### Example configuration using memberlist, boltdb-shipper, and S3 for storage

```yaml
loki:
  structuredConfig:
    ingester:
      # Disable chunk transfer which is not possible with statefulsets
      # and unnecessary for boltdb-shipper
      max_transfer_retries: 0
      chunk_idle_period: 1h
      chunk_target_size: 1536000
      max_chunk_age: 1h
    storage_config:
      aws:
        s3: s3://eu-central-1
        bucketnames: my-loki-s3-bucket
      boltdb_shipper:
        shared_store: s3
    schema_config:
      configs:
        - from: 2020-09-07
          store: boltdb-shipper
          object_store: aws
          schema: v11
          index:
            prefix: loki_index_
            period: 24h
```

The above configuration selectively overrides default values found in the `loki.config` template file.

Using `loki.structuredConfig` it is possible to externally set most any configuration parameter (special considerations for elements of an array).

```
helm upgrade loki-distributed --install -f values.yaml --set loki.structuredConfig.storage_config.aws.bucketnames=my-loki-bucket
```

`loki.config`, `loki.schemaConfig` and `loki.storageConfig` may also be used in conjuction with `loki.structuredConfig`. Values found in `loki.structuredConfig` will take precedence. Array values, such as those found in `loki.schema_config` will be overridden wholesale and not amended to.

For `loki.schema_config` its generally expected that this will always be configured per usage as its values over time are in reference to the history of loki schema versions and schema configurations throughout the lifetime of a given loki instance.

Note that when using `loki.config` must be configured as string.
That's required because it is passed through the `tpl` function in order to support templating.

When using `loki.config` the passed in template must include template sections for `loki.schemaConfig` and `loki.storageConfig` for those to continue to work as expected.

Because the config file is templated, it is also possible to reference other values provided to helm e.g. externalize S3 bucket names:

```yaml
loki:
  config: |
    storage_config:
      aws:
        s3: s3://eu-central-1
        bucketnames: {{"{{"}} .Values.bucketnames {{"}}"}}
```

```console
helm upgrade loki-distributed --install -f values.yaml --set bucketnames=my-loki-bucket
```

## Gateway

By default and inspired by Grafana's [Tanka setup](https://github.com/grafana/loki/tree/master/production/ksonnet/loki), the chart installs the gateway component which is an NGINX that exposes Loki's API
and automatically proxies requests to the correct Loki components (distributor, querier, query-frontend).
The gateway must be enabled if an Ingress is required, since the Ingress exposes the gateway only.
If the gateway is enabled, Grafana and log shipping agents, such as Promtail, should be configured to use the gateway.
If NetworkPolicies are enabled, they are more restrictive if the gateway is enabled.

## Metrics

Loki exposes Prometheus metrics.
The chart can create ServiceMonitor objects for all Loki components.

```yaml
serviceMonitor:
  enabled: true
```

Furthermore, it is possible to add Prometheus rules:

```yaml
prometheusRule:
  enabled: true
  groups:
    - name: loki-rules
      rules:
        - record: job:loki_request_duration_seconds_bucket:sum_rate
          expr: sum(rate(loki_request_duration_seconds_bucket[1m])) by (le, job)
        - record: job_route:loki_request_duration_seconds_bucket:sum_rate
          expr: sum(rate(loki_request_duration_seconds_bucket[1m])) by (le, job, route)
        - record: node_namespace_pod_container:container_cpu_usage_seconds_total:sum_rate
          expr: sum(rate(container_cpu_usage_seconds_total[1m])) by (node, namespace, pod, container)
```

## Caching

The chart can configure up to four Memcached instances for the various caches Lokis can use.
Configuration works the same for all caches.
The configuration of `memcached-chunks` below demonstrates setting additional options.

Exporters for the Memcached instances can be configured as well.

```yaml
memcachedExporter:
  enabled: true
```

### memcached-chunks

```yaml
memcachedChunks:
  enabled: true
  replicas: 2
  extraArgs:
    - -m 2048
    - -I 2m
    - -v
  resources:
    requests:
      cpu: 500m
      memory: 3Gi
    limits:
      cpu: "2"
      memory: 3Gi

loki:
  config: |
    chunk_store_config:
      chunk_cache_config:
        memcached:
          batch_size: 100
          parallelism: 100
        memcached_client:
          consistent_hash: true
          host: {{"{{"}} include "loki.memcachedChunksFullname" . {{"}}"}}
          service: memcached-client
```

### memcached-frontend

```yaml
memcachedFrontend:
  enabled: true

loki:
  config: |
    query_range:
      cache_results: true
      results_cache:
        cache:
          memcached_client:
            consistent_hash: true
            host: {{"{{"}} include "loki.memcachedFrontendFullname" . {{"}}"}}
            max_idle_conns: 16
            service: memcached-client
            timeout: 500ms
            update_interval: 1m
```

### memcached-index-queries

```yaml
memcachedIndexQueries:
  enabled: true

loki:
  config: |
    storage_config:
      index_queries_cache_config:
        memcached:
          batch_size: 100
          parallelism: 100
        memcached_client:
          consistent_hash: true
          host: {{"{{"}} include "loki.memcachedIndexQueriesFullname" . {{"}}"}}
          service: memcached-client
```

### memcached-index-writes

NOTE: This cache is not used with `boltdb-shipper` and should not be enabled in that case.

```yaml
memcachedIndexWrite:
  enabled: true

loki:
  config: |
    chunk_store_config:
      write_dedupe_cache_config:
        memcached:
          batch_size: 100
          parallelism: 100
        memcached_client:
          consistent_hash: true
          host: {{"{{"}} include "loki.memcachedIndexWritesFullname" . {{"}}"}}
          service: memcached-client
```

## Compactor

Compactor is an optional component which must explicitly be enabled.
The chart automatically sets the correct working directory as command-line arg.
The correct storage backend must be configured, e.g. `s3`.

```yaml
compactor:
  enabled: true

loki:
  config: |
    compactor:
      shared_store: s3
```

## Ruler

Ruler is an optional component which must explicitly be enabled.
In addition to installing the ruler, the chart also supports creating rules.
Rules files must be added to directories named after the tenants.
See `values.yaml` for a more detailed example.

```yaml
ruler:
  enabled: true
  directories:
    fake:
      rules.txt: |
        groups:
          - name: should_fire
            rules:
              - alert: HighPercentageError
                expr: |
                  sum(rate({app="loki"} |= "error" [5m])) by (job)
                    /
                  sum(rate({app="loki"}[5m])) by (job)
                    > 0.05
                for: 10m
                labels:
                  severity: warning
                annotations:
                  summary: High error percentage
```