Logs and the agent

Two ways to get logs in, both authenticated with an ingest key (ksi_..., see site key and ingest key):

• The knockstat agent tails log files, Docker containers or Kubernetes pod logs and ships each request. It remembers where it stopped, so a restart loses and repeats nothing.
• Vector or Fluent Bit, if you already run one: point its HTTP output at POST /v1/logs.

One log source belongs to one site, because the ingest key picks the site. Use one key per site.

Log formats

Name the format of each source. A line that does not parse is counted and skipped, never guessed.

A format with no host leaves request.host empty. For a log that serves one site, set the source’s host in the agent; for several sites in one log, use a format with the host. The client address is read from the line and never stored, so put the real client there: with nginx behind a proxy, use set_real_ip_from and real_ip_header; behind Cloudflare, caddy_json and nginx_json read Cf-Connecting-Ip for you.

nginx, with the host

log_format knockstat '$host $remote_addr - $remote_user [$time_local] '
                     '"$request" $status $body_bytes_sent '
                     '"$http_referer" "$http_user_agent"';   # combined_host

access_log /var/log/nginx/access.log knockstat;

nginx, JSON

log_format knockstat_json escape=json
  '{'
    '"msec":"$msec",'
    '"time_iso8601":"$time_iso8601",'
    '"remote_addr":"$remote_addr",'
    '"http_x_forwarded_for":"$http_x_forwarded_for",'
    '"http_cf_connecting_ip":"$http_cf_connecting_ip",'
    '"host":"$host",'
    '"request_method":"$request_method",'
    '"request_uri":"$request_uri",'
    '"server_protocol":"$server_protocol",'
    '"status":"$status",'
    '"body_bytes_sent":"$body_bytes_sent",'
    '"http_referer":"$http_referer",'
    '"http_user_agent":"$http_user_agent"'
  '}';

access_log /var/log/nginx/access.json knockstat_json;

Caddy

example.com {
    log {
        output file /var/log/caddy/access.log
        format json
    }
}

Traefik

Traefik drops request headers from its access log by default; keep the ones knockstat reads:

# traefik.yml (static config)
accessLog:
  filePath: /var/log/traefik/access.log   # omit to log to stdout
  format: json
  fields:
    headers:
      defaultMode: drop
      names:
        User-Agent: keep
        Referer: keep
        X-Forwarded-For: keep     # if Traefik sits behind an internal load balancer
        Cf-Connecting-Ip: keep    # if Cloudflare is in front

The agent

Config

The agent reads /etc/knockstat/agent.toml (or the path you pass with --config). Keep the ingest key out of the file: set KNOCKSTAT_INGEST_KEY in the environment.

ingest_url = "https://app.knockstat.com"            # base URL; the agent posts to /v1/batch
state_dir = "/var/lib/knockstat-agent"   # one checkpoint file per source
# ingest_key = "ksi_..."                 # or env KNOCKSTAT_INGEST_KEY (preferred)

[[sources]]
id = "nginx"                             # stable: a new id starts the source over
path = "/var/log/nginx/access.log"
format = "combined"                      # see Log formats
host = "www.example.com"                 # fills request.host for lines that carry none
start_at = "end"                         # "end" (default) or "beginning"; first run only

Other source kinds: kind = "docker" reads container logs through the Docker Engine API (select containers by containers, labels or all_containers); kind = "kubernetes" reads pod logs from the node (select by namespaces, pods and containers, with * wildcards). Both take the application’s own format, such as traefik_json, not the container wrapper. Docker access is root-equivalent on the host, so treat the agent’s container accordingly.

Run it with knockstat-agent --config /etc/knockstat/agent.toml; add --once to ship what is there and exit.

Docker

The image is a single static binary (about 4 to 5 MB) running as user 10001.

docker run -d --name knockstat-agent --read-only --cap-drop ALL \
  -e KNOCKSTAT_INGEST_KEY=ksi_... \
  -v /etc/knockstat/agent.toml:/etc/knockstat/agent.toml:ro \
  -v /var/log/nginx:/logs:ro \
  -v knockstat-agent-state:/var/lib/knockstat-agent \
  ghcr.io/erphq/knockstat-agent:latest

In agent.toml the source path is then /logs/access.log. The default user must be able to read the log: add --group-add 4 for Debian and Ubuntu nginx logs (group adm), or run as root for a root-only log. Arguments replace the default command, so repeat --config /etc/knockstat/agent.toml if you pass any.

Binary and .deb

Download the static binary or the .deb from knockstat.com/download/agent.

# static binary (x86_64 and aarch64 Linux, nothing to install)
install -m 0755 knockstat-agent-<version>-x86_64-unknown-linux-musl /usr/local/bin/knockstat-agent
knockstat-agent --config /etc/knockstat/agent.toml

# Debian or Ubuntu with systemd
dpkg -i knockstat-agent_<version>_amd64.deb
$EDITOR /etc/knockstat/agent.toml                    # ingest_url and at least one [[sources]]
install -m 0600 /dev/null /etc/knockstat/agent.env
echo 'KNOCKSTAT_INGEST_KEY=ksi_...' > /etc/knockstat/agent.env
systemctl enable --now knockstat-agent
journalctl -u knockstat-agent -f

The package creates a knockstat-agent user in group adm, which reads /var/log/nginx on Debian and Ubuntu. The service is not enabled on install, because it needs a source and a key first.

Kubernetes (Helm)

The chart runs one agent pod per node as a DaemonSet. Create a Secret with the ingest key, then install the chart with your sources:

kubectl create secret generic knockstat-agent-ingest --from-literal=ingest-key=ksi_...

helm install knockstat-agent ./knockstat-agent -f values.yaml

# values.yaml: read Traefik's JSON access log on k3s, from the pods' stdout
ingestUrl: "https://app.knockstat.com"
sources:
  - id: k3s-traefik
    kind: kubernetes
    format: traefik_json
    namespaces: [kube-system]
    pods: ["traefik-*"]
    containers: [traefik]
    startAt: end

A kind: kubernetes source reads pod logs under /var/log/pods on each node and tags every event with $k8s_namespace, $k8s_pod, $k8s_container and $k8s_node. It needs no RBAC. Keep the selectors narrow: lines not in the format are counted as unparsed. To read a log file instead, mount the host directory with hostLogMounts and point a source at it. The key Secret name defaults to knockstat-agent-ingest with key ingest-key (ingestKeySecret).

How the agent delivers

• At least once, deduplicated. Each event’s id comes from its position in the log, so a restart or a resend is stored once.
• Checkpoint after acknowledgement. The agent moves on only after ingest answers 202. While ingest is down it stops and rereads from the same offset, so the log file is the buffer; keep your log retention longer than an outage.
• Backoff. 429, 503 and network errors retry with backoff, honouring Retry-After. A 401 retries every 60 seconds and never advances: check the key.
• Rotation. Rename-and-recreate rotation (logrotate default, Caddy) is followed. Avoid copytruncate, which repeats offsets.
• Permissions. The agent needs read access to each log file and write access to its state directory, nothing else.

Vector and Fluent Bit

If you already run Vector or Fluent Bit, send the log lines to ingest instead of running the agent. Three things go in every config: the ingest key as a bearer token, the log format in ?format=, and the shipper’s envelope in ?envelope=.

Use the plain base format: both shippers already strip the container wrapper and join split lines. Write a + in a format as %2B. Send at most 5,000 records per request and gzip the body. Fluent Bit’s msgpack and gelf output is not accepted; use json or json_lines.

Vector

# /etc/vector/vector.yaml: nginx on a host or VM
data_dir: /var/lib/vector

sources:
  nginx:
    type: file
    include: ["/var/log/nginx/access.log"]
    offset_key: offset          # position-based ids, so a re-read is not double-counted

sinks:
  knockstat:
    type: http
    inputs: ["nginx"]
    uri: "https://app.knockstat.com/v1/logs?format=combined_host&envelope=vector"
    method: post
    compression: gzip
    encoding:
      codec: json
    framing:
      method: newline_delimited
    auth:
      strategy: bearer
      token: "${KNOCKSTAT_INGEST_KEY}"   # ksi_..., from the environment
    batch:
      max_events: 1000            # at most 5000

For Kubernetes, use the kubernetes_logs source (it removes the container wrapper itself), select the controller with extra_label_selector, and set format=ingress_nginx or whichever format your controller writes. Tested with Vector 0.46.1.

Fluent Bit

[SERVICE]
    Flush          1

[INPUT]
    Name           tail
    Path           /var/log/nginx/access.log
    DB             /var/lib/fluent-bit/nginx.db
    Read_from_Head On

[OUTPUT]
    Name           http
    Match          *
    Host           app.knockstat.com
    Port           443
    tls            On
    URI            /v1/logs?format=combined_host&envelope=fluentbit
    Format         json_lines
    Header         Authorization Bearer ${KNOCKSTAT_INGEST_KEY}
    compress       gzip

For Kubernetes, add multiline.parser cri to the tail input, which removes the wrapper and joins split lines, and use the plain format in the URL. Do not set Offset_Key: after a restart the offset is not a stable position. Tested with Fluent Bit 4.0.3.

Every request answers 202 {"accepted": n, "rejected": [...]}. A line that does not parse is rejected on its own and the rest are stored. A resent line is stored once. Lines in a format with no host (combined, traefik, ingress_nginx) have an empty host here, so prefer a format that logs it.

Stuck? Email [email protected].