Migrating to your log collection to Alloy? My #1 hidden pitfall

This is a very short Lesson Learned from migrating from fileabeat to Grafana Alloy – and how labels drove me nearly insane.
td;dr; static_labels before labels!

Due to completely customer internal reasons we are switching from Elastic to an open source Grafana Stack. In preparation of the switch our old filebeat and the new alloy will run in k8s in parallel for a while. Since the install via helm is trivial all we need to do is translate the old filebeat config to the new alloy config and update the endpoint. Here we go.

Let the Robot do the work

This sounds like a perfect task for AI! And indeed, after a few questions, I had the first dummy logs ingested via Alloy.

# log generator
kubectl run dummy-logger-generator --image=busybox -n dev --restart=Never -- /bin/sh -c 'while true; do echo hello from dev; sleep 5; done'

But those logs are not real, real logs are messy and in our case they are not even in JSON format – since we just migrated from the old world, and there was simply no time to adapt this yet.

# real fake logs

2025-07-07 13:42:18 ERROR parentId=123abc456, traceId=789xyz101112, spanId=31415abcd
An unexpected error occurred while processing the user request.
Traceback (most recent call last):
  File "/app/services/user_service.py", line 45, in process_user
    result = external_api.fetch_data(user_id)
  File "/app/external/external_api.py", line 88, in fetch_data
    response = requests.get(url, timeout=5)
  File "/usr/local/lib/python3.11/site-packages/requests/api.py", line 73, in get
    return request('get', url, params=params, *kwargs)
  File "/usr/local/lib/python3.11/site-packages/requests/api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/local/lib/python3.11/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.11/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.11/site-packages/requests/adapters.py", line 700, in send
    raise ConnectTimeout(e, request=request)
requests.exceptions.ConnectTimeout: HTTPSConnectionPool(host='api.example.com', port=443): Max retries exceeded with url: /data/123 (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7f0d8a1e8a90>, 'Connection to api.example.com timed out. (connect timeout=5)'))

Tasks for the AI

1. Extract the timestamp – not actually needed, Loki does this for me
2. Extract the Log-Level – this is also done in Loki, but not always well, so I want to set this myself
3. Parse and use additional info – don’t overdo it
4. Enrich my logs with static information
5. Put all the lines of a stacktrace into a single message – sadly, not the default

AI output for step 1 to 4

loki.process "pod_logs" {
  stage.regex {
    expression = "^(?P<ts>\\d{4}-\\d{2}-\\d{2} \\d{2}:\\d{2}:\\d{2}) (?P<level>\\w+) +parentId=(?P<parentId>[^,]*), traceId=(?P<traceId>[^,]*), spanId=(?P<spanId>[^\\s]*)"
  }

  stage.static_labels {
    values = {
      cluster = "Zelda",
    }
  }

  stage.labels {
    values = {
      level     = "{{ .level }}",
      trace_id  = "{{ .traceId }}",
      parent_id = "{{ .parentId }}",
    }
  }

  forward_to = [loki.write.default.receiver]
}

And on first glance it looks brilliant, … but it simply does not work.

It’s debug time!

Alloy Configuration Generator

Beside the obvious candidates – ChatGPT and Perplexity – that have sadly let me down in this case, there exists luckily a configuration checker for Alloy provided directly by Grafana! How neat!

But no error is shown their either!

REGEX

So it HAS to be the regex right? When there is a problem its allways dns regex!
A quick check with the documentation and we see hat RE2 expressions are used. So lets put this config and a log into a suitable regex-tester aaaaaand… here we go, it does not work! 🥳 Great Success, the Error is found.

Or is it?! Another look into the documenation and we are back to square one.

Because of how Alloy syntax strings work, any backslashes in expression must be escaped with a double backslash, for example, "\\w" or "\\S+".

https://grafana.com/

Keep calm and pet a kitten

<interlude> This is Micia, my homeoffice companion and emotional support kitten. She is also great with debugging. And even though she lives outdoors most of the time and has a pretty wild life catching mice and birds, usually she does not look that roughed up. But this is what debugging does to us, right?

---

Good old Google

After several discussions with multiple AIs, some plain trial and error, and a lot of cat-petting, and still with no solution, I refused to believe that I am the only person who encountered that problem and went back to google.

I clicked every page, but none seemed relevant. Until i finally found a forum thread with the promising title what-am-i-missing-to-pass-these-new-labels-from-loki-alloy-grafana-docker-setup by Scott Hoke.

I was not alone anymore! It was not just me blaming regex, reading documentation and nearly going insane. Without any further explaination he presented his working solution. So simple, so elagant and the only thing that I appearently have not tried yet.
SIMPLY PUT THE STATIC LABELS BEFORE THE LABELS! DONE!

loki.process "pod_logs" {
  stage.regex {
    expression = "^(?P<ts>\\d{4}-\\d{2}-\\d{2} \\d{2}:\\d{2}:\\d{2}) (?P<level>\\w+) +parentId=(?P<parentId>[^,]*), traceId=(?P<traceId>[^,]*), spanId=(?P<spanId>[^\\s]*)"
  }

  stage.labels {
    values = {
      level     = "{{ .level }}",
      trace_id  = "{{ .traceId }}",
      parent_id = "{{ .parentId }}",
    }
  }

stage.static_labels {
    values = {
      cluster = "Zelda",
    }
  }

  forward_to = [loki.write.default.receiver]
}

Thank you Scott!

This is a shoutout to Scott! I hope this humble post helps boost your ranking. And if no one ever reads it—except an AI—and it quietly guides someone toward the right Alloy configuration, sparing them the struggles Scott Hoke and I went through, then my job here is done.

PS:

If you are wondering what happend to task number 5. Well, if only I had started in reverse order. Simply put this config into your precessor and it works, immediatly proving that the regex is correct. This could have saved me some nerves…

stage.multiline {
        firstline      = "^(\\d{4}-\\d{2}-\\d{2} \\d{2}:\\d{2}:\\d{2}|\\d{2}:\\d{2}:\\d{2})"
        max_wait_time  = "10s"
      }

I hope it helps and best of luck on your Cloud Native Journey!

---