What is Vector

Vector is a tool that takes metrics/events/logs data, routes and manipulates them and sends them to sinks. In that it is similar to software like Logstash or Riemann (and can mostly replace them)

It is made to serve as middleman in observability pipeline, applying any required transformations on the incoming streams of logs, metrics or traces

Why would I need such a tool?

The range is very wide due to how generic it is but the use cases I found particularly useful:

• parsing system logs into fields - while it is best done on the source devices it is not always possible (say logs from switches/routers)
• splitting input stream into different targets - for example sending logs related to mail or network operation to a different elasticsearch index, or saving audit logs in a different store
• garnishing entries with additional data - like adding GeoIP data to firewall logs so the end application can do geolocation based queries directly on database
• adding/dropping/reformatting/editing fields - sometimes you might not care about all fields, want them merged or split, or anonymized

That sounds like Logstash. Why would I use it instead of Logstash?

Ops-wise it has smaller memory footprint and better performance.

Language-wise, explicit source -> destination links instead of treating everything as one big pipeline like in Logstash are more readable.

It also handles metrics natively while in Logstash it is limited.

Vector basics

Sources, sinks, and transforms

In Vector configuration language flow of data is basically a DAG from source thru zero or more transforms to the sink.

1sources:
2  vector_metrics:
3    type: internal_metrics
4    scrape_interval_secs: 10
5  syslog_remote:
6    type: socket
7    address: "0.0.0.0:3515"
8    mode: tcp
9    max_length: 102400

In this example we gather internal metrics under vector_metrics source and receive JSON-encoded syslog entries. We will decode them in the next step.

 1transforms:
 2  parse_syslog:
 3    type: remap
 4    inputs: ["syslog_remote"]
 5    reroute_dropped: true
 6    source: |
 7      v, err = parse_json(string!(.message))
 8      if err != null {
 9        abort "bad json: " + err
10      }
11      . = v      

This is done so we have fine-grained control on the error handling. The dropped messages are routed to parse_syslog.dropped if reroute_dropped: true is set, that can be used to debug dropped events. Abort passes unmodified event so any modifications are lost.

If you are 100% sure JSON will always be correct you can just add:

1   decoding:
2     codec: json

to the source config to decode it directly.

You can then display it in console for debug (make sure it isn't reading its own logs on input else it will re-emit its own logs in a loop!):

1sinks:
2  console_out:
3    type: console
4    inputs:
5      - parse_syslog
6    encoding:
7      codec: "json"
8      json:
9        pretty: true

and see messages emitted on stderr/systemd logs, example being a firewall log from Linux machine:

1 vector[362335]: {
2 vector[362335]:   "@timestamp": "2026-07-15T19:52:49.760181+02:00",
3 vector[362335]:   "facility": "kern",
4 vector[362335]:   "fromhost": "dc1-example-vpn1",
5 vector[362335]:   "message": " [17998932.400138] vsa[jkowalski-out]: IN=tun2 OUT=eth0 MAC= SRC=192.168.1.160 DST=10.0.1.2 LEN=64 TOS=0x00 PREC=0x00 TTL=63 SPT=55218 DPT=3128 WINDOW=65535 RES=0x00 CWR ECE SYN URGP=0 ",
6 vector[362335]:   "priority": "info",
7 vector[362335]:   "program": "iptables",
8 vector[362335]:   "route": "iptables"
9 vector[362335]: }

or send it to Elasticsearch for further viewing/processing

 1sinks:
 2  syslog:
 3    type: elasticsearch
 4    inputs:
 5      - parse_syslog
 6    mode: bulk
 7    bulk:
 8      index: syslog-%Y.%m.%d
 9    endpoints:
10      - http://127.0.0.1:9200

Enrichment tables

Those are basically a way to look up a value based on another one. Example for GeoIP:

1
2enrichment_tables:
3    geoip_city:
4      type: geoip
5      path: "/var/lib/geoip/GeoLite2-City.mmdb"
6    geoip_asn:
7      type: geoip
8      path: "/var/lib/geoip/GeoLite2-ASN.mmdb"

then, in transform, you feed it the IP to resolve

 1    geo_city, err = get_enrichment_table_record("geoip_city", { "ip": .src_ip })
 2    if err == null {
 3        .geoip = geo_city
 4        del(.geoip.timezone)
 5        geo_asn, err = get_enrichment_table_record("geoip_asn", { "ip": .src_ip })
 6        if err == null {
 7          .as.number = geo_asn.autonomous_system_number
 8          .as.org = geo_asn.autonomous_system_organization
 9          .as.net = geo_asn.network
10        }
11    }

This example takes src_ip field (assumed existence; if the field doesn't exist it is better to put whole block in if case for that) and:

• looks up the GeoIP city data
• removes timezone (we don't care about keeping that data as it is easy to look up)
• looks up the GeoIP ASN data
• cherry picks the fields we want in database (AS number/org and network)

Inputs and routing

Every source and transform produces one or more outputs that can then be used in another transform or sink in inputs fields.

For example the opentelemetry source provides .logs/.traces/.metrics and transforms like route can split the incoming stream into multiple outputs:

1transforms:
2  route:
3    inputs: ["parse_syslog"]
4    type: route
5    route:
6      ok: '!exists(.route)'
7      iptables: '.route == "iptables"'
8      dhcpd: '.route == "dhcpd"'
9      mail: '.route == "mail"'

Note that route type routes non-exclusively - if an event matches more than one route, it is sent to multiple output streams - for exclusive "one event one route" behavior there is exclusive_route transform.

The route transform also emits _unmatched output for events that did not match any route.

Another output worth noting is dropped, which emits events that were aborted in a transform when reroute_dropped: true is set (note the opposite defaults: reroute_unmatched defaults to true, reroute_dropped to false).

Example config

Here is whole example.

The parts of it are explained below:

Anonymizing Claude Code AI traces

Imagine your company wants to track the Claude Code usage but without giving management too intrusive a view into users' usage patterns. What we need to do is to take a set of fields that identify a user and anonymize them.

Note that it CANNOT be a simple hash (which would be easy to do on many OTLP-compatible collectors), because if it is a simple hash it is trivial to do email -> hash map if you just know someone's corporate email. We have to use a cryptographic function with a secret to map it into something that can't be trivially reverted. So we use HMAC with a private key.

First, basic setup of sources:

 1api:
 2  enabled: true
 3  address: "127.0.0.1:8686"
 4secret:
 5  file:
 6    type: file
 7    path: /etc/vector/secrets.json # {"anon_key": "anonymization secret"}
 8
 9sources:
10  vector_metrics: # keep the internal metrics available for scrape
11    type: internal_metrics
12    scrape_interval_secs: 10
13  otel_in:
14    type: opentelemetry
15    grpc:
16      address: 0.0.0.0:14317
17    http:
18      address: 0.0.0.0:14318
19    # that means "decode in OTLP native format, not in vector native format, we need it as
20    # we will be pushing it to OTLP-compatible store   
21    use_otlp_decoding:
22      logs: true
23      metrics: true
24      traces: true

The only thing worth noting here is usage of secret; an environment variable could be used but that is not reloaded when vector does graceful config reload, while secret store is. We want that feature because we might also want to rotate the secret so as not to connect user ID with a given hash permanently.

Then, the tricky step. OTLP format is a bit verbose, for example instead of having key -> value map, it puts resource attributes into array (despite spec saying they are unique!) and doesn't put value directly but has type -> value indirection:

1 "resource": {
2        "attributes": [
3          {
4            "key": "service.name",
5            "value": {
6              "stringValue": "my.service"
7            }
8          }
9        ]

so we have to do some looping to actually find and remove the value. We also need to do a separate operation for traces/metrics/logs due to different field naming and the fact that the OTLP sink is a pretty thin wrapper that outputs data directly to a given URL (so we need one per type). Example for metrics:

 1transforms:
 2  anonymize_metrics:
 3    type: remap
 4    inputs:
 5      - otel_in.metrics  # metrics arrive on the .metrics output as log events
 6    source: |
 7      key = "SECRET[file.anon_key]"
 8      # list of keys we want to anonymize
 9      anon_keys = ["user.email", "user.id", "user.account_id", "user.account_uuid"]
10      # and ones we don't care about
11      del_keys  = ["os.version"]
12
13      # due to language being compiled we can't get too dynamic with programming here
14      .resourceMetrics = map_values(array!(.resourceMetrics)) -> |rm| {
15        r = object!(rm)
16        if exists(r.resource.attributes) {
17          r.resource.attributes = filter(array!(r.resource.attributes)) -> |_index, attr| {
18            !(is_object(attr) && includes(del_keys, attr.key))
19          }
20        }
21        r
22      }
23
24      # now we modify the fields
25      . = map_values(., recursive: true) -> |value| {
26        if is_object(value) && includes(anon_keys, value.key) {
27          attr = object!(value)
28          attr.value.stringValue =
29            slice!(encode_base16(hmac(string!(attr.value.stringValue), key, algorithm: "SHA-256")), 0, 10)
30          attr
31        } else {
32          value
33        }
34      }      

At this point we have anonymize_metrics transform that takes and emits OTLP-compatible structure, so we only need to send it:

 1  otel_out_metrics:
 2    type: opentelemetry
 3    inputs:
 4      - anonymize_metrics
 5    protocol:
 6      type: http
 7      uri: "http://127.0.0.1:4318/v1/metrics"
 8      method: post
 9      encoding:
10        codec: otlp

and repeat the exercise for remaining 2 types (logs/traces). All is in the provided example file above.

Converting OTLP metrics to Prometheus ones

This is very simple. All you need to do is to use OTEL source in native mode:

1sources:
2  otel_raw_in: # loopback in for prometheus conversion
3    type: opentelemetry
4    use_otlp_decoding: false # this makes it use Vector format for outgoing data instead of OTLP native decoding
5    grpc:
6      address: 127.0.0.1:14327
7    http:
8      address: 127.0.0.1:14328

and just send it to prometheus compatible sink:

 1  prometheus_out:
 2    type: prometheus_remote_write
 3    inputs:
 4      - otel_raw_in.metrics
 5    endpoint: https://promwriter-api.example.com/api/v1/write
 6    expire_metrics_secs: 36000
 7    healthcheck: # you might need to turn it off if you use backend that doesn't have Prometheus healthcheck endpoint, like VictoriaMetrics
 8      enabled: false
 9    batch:
10      max_events: 1000
11      timeout_secs: 5

Vector-native format is WAY easier to anonymize (you just have key-value in that format), but if you have the setup above for anonymization you can loop it back to itself so the same vector instance is doing double duty.

 1  otel_out_loop:
 2    type: opentelemetry
 3    inputs:
 4      - anonymize_metrics
 5    protocol:
 6      type: http
 7      uri: "http://127.0.0.1:14328/v1/metrics"
 8      method: post
 9      encoding:
10        codec: otlp

This is mostly needed as there is no native function to parse it to/from OTLP format.

Plugging Claude into it

For any production workload put the whole thing behind loadbalancer preferable also with some form of auth (you can use OTEL_EXPORTER_OTLP_HEADERS for simple auth in Claude Code), required settings in ~/.claude/settings.json look like this:

 1{
 2  "env": {
 3    "CLAUDE_CODE_ENABLE_TELEMETRY": "1",
 4    "CLAUDE_CODE_ENHANCED_TELEMETRY_BETA": "1",
 5    "OTEL_LOG_TOOL_DETAILS": "0",
 6    "OTEL_LOG_USER_PROMPTS": "0",
 7    "OTEL_TRACES_EXPORTER": "otlp",
 8    "OTEL_METRICS_EXPORTER": "otlp",
 9    "OTEL_LOGS_EXPORTER": "otlp",
10    "OTEL_EXPORTER_OTLP_PROTOCOL": "http/protobuf",
11    "OTEL_EXPORTER_OTLP_ENDPOINT": "http://vector.example.com:14318",
12    "OTEL_EXPORTER_OTLP_HEADERS": "x-api-key=auth",
13    "OTEL_METRIC_EXPORT_INTERVAL": "60000"
14  }
15}

or you can use env with it

So after recent kerfuffle with Minio effectively deprecating OSS offering to focus on extracting money from AI bubble many hobbyist and small companies alike were left looking for solutions.

Personally I used it in few places, from home backup to build cache for CI/CD and few smaller work tasks so I will be evaluating replacement for these categories:

• ease of management from CI/CD - minio here is very friendly, you can generate any user/password and just set it so migrating from any existing system is pretty easy
• ease of setup - this is not looking from perspective of big setups, for those Ceph+RADOS is pretty much the only sensible option.
• ease of migration
• features related to backup safety like permissions or ILM features
• multi-site capability
• monitoring
• chance of getting rug-pulled again

What not be evaluted:

• web hosting features

The setup I initially used under minio was per-backup user that had read/write permissions + separately set ILM rules that kept the deleted files for a month so in case of encryption ransomware attack or any other case where attacker gets the backup credentials and starts to mess stuff up. The server was then replicated by replication builtin into minio into server at OVH, both being essentially independent servers.

Background

For years, I used Bareos, Bacula fork that was basically started for reason of "well, Bacula is too corporate and keeps the features in enterprise versions for years before putting it in community edition.

It was a very convenient system, as list of all backups of all my machines (half a dozen at home, half a thousand at work)

Over the years it came a full circle, with Bareos stopping to even support packages in Debian distribution. Main driving reason for initial install was "I can just install bareos client and get on with backing up my machine", and bareos looked to be the "community chosen" fork, but with time "just install the pacakge" was no longer true and only provided packages were snapshots of dev branch that were often buggy to the point of needing to occasionally restart the daemon when backups stopped working. It's "everything is a tape in disguise" approach was also problematic when trying to do anything but plain hard drive/tape backup

So I started looking for alternatives. There is no really good "enterprise" solution compared to bareos in OSS space, there is BackupPC but managing an Perl app that had last stable release in 4 years didn't looked like something I want to do in my free time. So I started to looking for alternatives

Rejected alternatives

I looked for something that could at least store data on S3 (due to ease of doing it on wider scale, and having cloud options if needed) and typical nice-to-haves of backup software like retention periods and ability to easily exclude files, preferably by means of just putting a file defining a dir/files to exclude

Borg backup

It's entirely fine and fitting. Sparse choice of storage backends removed it from the list

Syncthing

It's a great file synchronization tool; not designed for backups so it have light un-delete functionality; I already use it for sync but not exactly required feature-set for backups

Amanda

Just as with Bareos/Bacula, they are still in denial and pretend everything is tape. I used it some time ago and it was... okay but not something I was looking for. It also "decides for itself" when to make full or incremental based on storage which isn't exactly great when dealing with limited internet bandwidth..

Kopia

Lastly, the one most similar to Restic, Kopia. I test-drove the two for few months and my main take-away is restic is better for scripted backups while Kopia is better at "user UI first" approach. If all you need is to back-up few desktop machines, by all means just use Kopia, it does that excellent.

Why restic was chosen

• very good cli support
• always-incremental snapshot approach after every backup
• fast enough (within single digit % compared to kopia)
• Supports mounting snapshot as FUSE (few others like kopia or borg support that too, unlike bareos)
• Backup encryption by client (supported by most backup applications above)
• Available in Debian repository (borgbackup and amanda are available, kopia needs external repo)
• Multiple key support - ability to have "master" key that can access every archive and per-host keys that host can use to restore itself

EDIT: the problem with audio interface mentioned in article doesn't exist on recent version of pipewire and related alsa packages

My recently-bought Behringer UMC204HD audio interface, that is advertised as 2 inputs, 4 outputs, presents itself to Linux as one stereo input channel and one 4.0 output channel:

 1pacmd list-sinks |grep -P 'name: \<|channel map' -C 1
 2...
 3    index: 2
 4	name: <alsa_output.pci-0000_00_1b.0.iec958-stereo>
 5	driver: <module-alsa-card.c>
 6--
 7	sample spec: s16le 2ch 44100Hz
 8	channel map: front-left,front-right
 9	             Stereo
10...
11pacmd list-sources |grep -P 'name: \<|channel map' -C 1
12...
13    index: 2
14	name: <alsa_input.usb-BEHRINGER_UMC204HD_192k-00.analog-stereo>
15	driver: <module-alsa-card.c>
16--
17	sample spec: s32le 2ch 44100Hz
18	channel map: front-left,front-right
19	             Stereo
20
21...

I'm guessing the reason is plain "we just use some generic USB ADC/DAC chip and reference implementation comes in that config".

That poses a problem for most voice communication software, usually putting stereo input into it means it will sum both channels up, and summing a channel with signal to channel with zero signal results in volume drop (-6db usually).

Some software is competent enough to auto-gain out of that problem, but most don't even have VU meter, let alone any indication of sound level you're sending to other participants of the conversation so it's easy to be too quiet in out of the box config.

So we need to split one device into two. We might still occasionally want the stereo input so I won't try to replace it with two mono ones.

Why change clock speed

So why you would want to change speed of your I2C interface?

You might just need to probe your sensors more often.

You might also want to get around long standing I2C stretching bug, which caused that by default I2C stretching on rPi I2C controller is disabled.

in this post we will look at how systemd handles dependencies and how they can be used to increase the robustness of the system, and also discuss potential pitfalls.

For the existing services all of the examples should be put in /etc/systemd/system/<service you change>.d/<override>.conf, as this way you will only override the things you change in unit, instead of editing whole unit file in /lib (and having your changes overridden on package upgrade).

Unless mentioned otherwise the dependencies are specified in [Unit] section

Setting up our editor (CLion, VS Code) to debug the STM32 ARM chips.

See Part 1 on how to connect to board and Part 2 on how to setup project and OpenOCD for compile and debug

Setting up project and programming our first piece of Rust code onto microcontroller (Part1)

Preparing build environment

After we finished the hardware setup in Part 1 we should have working openocd connection.

Rest of this series of posts will assume stlink or jlink debugger + Blue Pill board. There is variety of clones like these:

but legality of the software on the clones is at very best dubious. You can also just get a Nucleo board which contains debugger and small eval board all in one

We will also be using template graciously provided by Rust Cortex-M team. Make sure to use at least Rust 1.53 (tutorial is built on that version) and install cross-compile dependencies of rust and binutils-arm-none-eabi:

Setting up tooling and development hardware for Rust on the ARM (STM32 CPUs)

What hardware do we need?

• Dev board. Just about any will do, Blue Pill is pretty popular, other option is one of the STM32 Discovery boards that come with the programmer.
• Programmer/debugger - any of the ST-Link (or clones) will do, here we will be using ST-Link V2 clone. Really, anything supported by openocd is fine, altho some need some tweaking
• Power supply for a dev board - a lot of them just accept micro-USB connector

Most dev boards come with some example code, the barebones often just have LED blinking, but I've seen some that even output on USB.

The more sophisticated ones that have extra peripherals usually come with demo of them so you might want to plug them before doing anything else to have a look.

Git has very convenient feature called filters. What it basically allows to do is to pipe the content of a given file (decided by entry in .gitattributes thru external filter command.

The different hook points are on file checkout (so called "smudge"), check-in, and on diff.

Here are two useful cases for using it:

Formatting files on checkin

If your language have a formatter that just accepts stdin and outputs formatted file on stdout, like Go, it is very straightforward to add it:

1git config --global filter.gofmt.clean "gofmt"
2git config --global filter.gofmt.smudge "gofmt" # or "cat" if you don't want to filter incoming commit

then in either ~/.config/git/attributes or in repository itself under .gitattributes

1*.go filter=gofmt

Do remember that if used on repo where others don't use formatter regularly you might get a bunch of spurious formatting modifications during daily work.Especially with filter on checkout.