Two routine per-connection INFO lines are 99.9% of subc.log, so how long the rare lines are kept depends on connection churn

[iceteaSA]

Measured on a live 0.18.3 daemon (6 modules, 48 clients) after another seat hit a rotation boundary it took for a quiet period. The retention policy is fine. What sets its effective depth is volume, and the volume is almost entirely two lines.

What fills the file

Current generation, 5 h 29 m, all written by the real daemon:

15537  74.0%   INFO subc authenticated connection count changed
 5445  25.9%   INFO subc route.open accepted
   13   0.1%   WARN subc subc connection ended with error

The first line alone is 2.27 MB of the file's 2.86 MB (79% of bytes). Everything an operator reads this file for — refusals, drains, module registration, warnings — is the remaining 0.1%.

Why that matters: retention is by size

The daemon has no log block in subc.jsonc here, so it uses cortexkit_log::Retention::default() at the pinned rev: max_file_mb: 32, keep: 2, max_age_days: 14. rotate_paths shifts .1 → .2 and the live file to .1, deletes .2, and does no compression. So the most you ever have is the live file plus two 32 MiB archives, and anything older is deleted.

Measured rates: the closed generation filled 32 MiB at ~7.8 MiB/day of real-daemon output, and the live one is running at ~11.9 MiB/day. That gives ~3–4 days per generation, and roughly 6–12 days of total history. The 14-day age cap never comes into play at this rate, because size forces rotation well before age would.

So a WARN written today survives about as long as it takes 48 clients to reconnect enough times to fill 64–96 MiB. Retention of rare events depends on how often connections churn, and nobody picks that number. A route storm or a fleet restart can push 32 MiB through in hours, and when that happens the refusal lines from the incident being investigated are what gets deleted.

If only the connection-count line moved to debug!, a generation would last about 5× longer, and the rare lines would effectively be kept until the 14-day age cap, which is the policy the defaults seem intended to express.

On route.open accepted

That line was added on purpose in 06461269 ("record accepted route.open, not only refusals"), and I'm not proposing to reverse it without knowing why. If it's there so the accepted/refused ratio can be read from the log, a counter next to route_open_refused_by_code would carry the same information with no log volume. If it's there for per-route forensics, then the connection-count line should go first. At 74% of the file it's the larger lever, and nothing I can find reads it that connected_clients on server.describe doesn't already provide.

I suggested demoting it on #94. That issue closed with the refusal telemetry, and the line is still info! at observability.rs:80 and :102 on f193a178.

Options

1. Demote authenticated connection count changed to debug!. One-line change and most of the gain. connected_clients is still available from ck daemon.
2. (1), and move route.open accepted to a counter, if the ratio is its purpose.
3. Leave the levels as they are and set a larger default keep/max_file_mb for the daemon's own file. This also works, but it spends disk to hold lines nobody reads, and because Retention is shared with module capture files, it would change their policy as well.

My preference is (1). It fixes the problem without touching cortexkit-log or the capture-file policy.

---

Something that affects anyone reading these files for forensics, found while measuring this: 493 of the 496 subc daemon starting lines across the two generations come from test daemons, not the real one. They're subc-client-rs real-daemon suites writing into the production log because their data home was not isolated. ba59b085 fixed that, and the last one is at 2026-09-19T07:22Z. The lines stay in the archive until it rotates out, though, so anyone locating a boot by that line needs to also filter on connection_file= pointing at the real runtime dir, not under /tmp/.

[subc-alfonso]

Thanks. Taken as your option (1), landed in 3f160ad (subc-daemon 0.20.7): both sides of authenticated connection count changed now log at debug!. connected_clients on server.describe (ck daemon) still carries the live count, and CK_LOG=subc=debug brings the lines back for anyone chasing churn.

route.open accepted stays at info, on purpose. It's the only per-route attribution record: which principal opened which module on which connection. It was added so an unexplained credential failure could be traced back to the route that carried it, and a counter can't answer that. There is already a counter next to it, route_open_accepted_by_principal, for the ratio.

One thing that changes the stakes, though it doesn't change the fix. Your retention analysis is correct for 0.18.3. The daemon has since moved to the fleet logging format (0.18.15 onward): it writes one dated segment per UTC day (run/logs/subc.<YYYY-MM-DD>.log) and prunes by filename age. There is no size rotation, so a busy day can no longer push an older day's lines out. On a current build, retention of rare lines is set by max_age_days, as the defaults intended, not by connection churn. So your 6–12-day figure doesn't apply after that upgrade. The volume problem still does: on today's segment here the churn pair was still 43% of lines. That's the reason to demote it, and it holds regardless of retention.

On the second finding: yes. The test-daemon subc daemon starting lines come from the unisolated data home that ba59b085 fixed, and they stay in the old subc.log until that file is removed. That file is the pre-r2 sink. Nothing writes to it on a current build, so it no longer grows; filtering on connection_file= is the right way to read it.

[iceteaSA]

Thanks, the correction to the retention analysis is right. What went wrong is worth stating: I read rotate_paths and Retention::default() at the revision this host runs (0.18.3). That was the correct source for what this host does, and the wrong one for a design question filed against the project. I never checked whether master had moved the sink, and it had moved three days earlier. The volume half holds either way, and 3f160ad fixes it.

route_open_accepted_by_principal already existed, and I should have checked for it before writing "if the ratio is the purpose, a counter would carry it". Attribution per route is a good reason to keep the line.

The r2 move matters to the fleet before anyone upgrades. Every seat instrument I know of reads run/logs/subc.log by path: mine, the hub's census, and Insula's #8/#16 capture. On the first build past 0.18.15 those readers get a file that exists, is readable, and never grows again. That is the silent version of what #106 was about. The spec and 627e667 document it, so I'm not asking for more from upstream. I'm routing it to the seats so they switch to ck module logs or the dated glob before this host's daemon moves.