Performance test kit (k6 + Kubernetes)

Load-test an API that runs as a container in Kubernetes, with k6 running as a Job inside the same cluster.

Two paths, one set of scenarios

	k6/run-test.ps1	scripts/deploy-and-test.ps1	scripts/run-cluster-test.ps1
Where k6 runs	docker compose, on your machine	a Kubernetes Job, in the cluster	a Kubernetes Job, in the cluster
Use it for	writing and debugging scenarios; "does the script work"	every number you intend to act on; CI and the bash wrappers	the same, but driven from prompts: build, deploy, run, then hand you the Grafana link
Target must be	reachable from a container (host.docker.internal, not localhost)	in-cluster DNS (http://api-service:8080/...)	in-cluster DNS by default
Observability	Grafana + Prometheus via compose: the k6 load test dashboard, plus results/<run>/ artefacts (see below)	results/<run>/ artefacts (see below)	the same artefacts, plus an in-cluster Grafana link pinned to the run

All three write results/<timestamp>-<type>/{k6.log,summary.json,meta.json}, so a local run and a cluster run are compared the same way.

k6/scripts/            scenarios (smoke, load, rate, stress, soak, spike, journey)
k6/scripts/lib/        shared env handling + the machine-readable summary
k6/tests/              node-runnable check of the summary library
k8s/base/              namespace-free manifests: Deployment, Service (kustomize)
k8s/overlays/local/    the single-node local overlay
k8s/app/Dockerfile     runtime-only API image, built from a `dotnet publish` folder
k8s/.generated/        per-run overlays, regenerated by the runner (gitignored)
k8s/observability/     in-cluster Prometheus + Grafana for cluster runs
scripts/               the Kubernetes runner, the interactive wizard, the observability
                       supervisor for reaching Grafana, the baseline gate, cleanup
results/               run artefacts (gitignored - these are evidence, not source)
baselines/             the accepted numbers a run is compared against

Quick start

Prerequisites: Docker, kubectl pointed at a cluster, PowerShell 7+ (pwsh) for the runners, and Node only if you want to run the library test.

# 1. A smoke test first, always. If this fails, nothing after it means anything.
pwsh -File scripts/deploy-and-test.ps1 `
  -TargetUrl http://api-service:8080/api/health `
  -TestType smoke

# 2. A load test with your own knobs, reusing the deployment that is already up.
pwsh -File scripts/deploy-and-test.ps1 `
  -TargetUrl http://api-service:8080/api/orders `
  -TestType load -TestOnly `
  -EnvVars TARGET_VUS=200,HOLD_DURATION=15m

# 3. Accept the result as the baseline, then gate later runs against it.
pwsh -File scripts/compare-summary.ps1 -Current results/latest.txt -Baseline baselines/load.json -Update
pwsh -File scripts/compare-summary.ps1 -Current results/latest.txt -Baseline baselines/load.json

# 4. Tear everything down (results/ is left alone).
bash scripts/cleanup.sh

Local authoring, no cluster:

pwsh -File k6/run-test.ps1 -TargetUrl http://host.docker.internal:5000/api/health -TestType smoke

Getting local images into the cluster

-Image defaults to sevice-api:local, built locally, which only works if the cluster can see it. The runner handles this where it can:

Cluster	Behaviour
Docker Desktop Kubernetes	Shares the Docker daemon: nothing to do.
kind	The runner detects a kind-* context and runs kind load docker-image for each locally-built image.
minikube	Same, via minikube image load. A loaded tag stays loaded: check kubectl get nodes -o jsonpath='{.items[*].status.images[*].names}' and pass one that is already there with -K6Image instead of rebuilding.
Anything remote	Push to a registry the cluster can reach, and pass a real tag or digest: -Image registry/api@sha256:...

Prefer an image your application's CI built and published. -ApiPath still exists for building from source, but it encodes another repository's directory layout and is scheduled for removal (P1 item 8 below).

To build from source without that guess, name the Dockerfile and its context:

# The publish-output image this kit ships: no SDK, no NuGet cache, and a build that
# never sees a private feed's credentials.
dotnet publish -c Release -o .build/finance-api
pwsh -File scripts/deploy-and-test.ps1 -TargetUrl http://api-service:8080/healthz `
  -TestType load -Image finance-api:local `
  -Dockerfile k8s/app/Dockerfile -BuildContext .build/finance-api `
  -BuildArg APP_DLL=ExchangePortal.Finance.Api.dll

-BuildContext defaults to the Dockerfile's own directory, which is the only default that can work for a self-contained Dockerfile; the runner warns by name when a COPY source is not in the context it was given. scripts/run-cluster-test.ps1 finds the publish folder and its APP_DLL for you.

One command, interactively

scripts/run-cluster-test.ps1 is the front door for a cluster run: it asks what it needs to know, builds the API image, deploys it, runs the preflight, runs the chosen scenario as a Job, and prints the Grafana link for that run. Start with the plan rather than the run:

# Show me what you would do - builds nothing, deploys nothing.
pwsh -File scripts/run-cluster-test.ps1 -TestType load -TargetPath /api/new-core/wallets -DryRun

# Then for real. Everything else comes from k6\.env and the defaults.
pwsh -File scripts/run-cluster-test.ps1 -TestType load -TargetPath /api/new-core/wallets

# A 4 GiB pod with 2 CPUs, from the publish folder that is already on disk.
pwsh -File scripts/run-cluster-test.ps1 `
  -TestType stress -TargetPath /api/new-core/requests/active `
  -CpuRequest 2 -CpuLimit 2 -MemoryRequest 4Gi -MemoryLimit 4Gi

It prompts for the test type (from the scenarios that actually exist), the endpoint, and - when it cannot find one - the Dockerfile or dotnet publish folder to build from. Anything supplied as a parameter is not asked for, and -NonInteractive turns every remaining question into an error, so the same script works unattended.

Three things it does that are worth knowing about, because each one removes a way a cluster run goes wrong:

• Credentials go into a Secret, not into the Job. Whatever Protect-PerfTestSecretValues considers credential-shaped is sent to -CredentialsSecret and injected with envFrom; the rest becomes ordinary -EnvVars knobs. results/<run>/k6-job.yaml is the exact manifest that was applied, so this is the difference between a password in a file people attach to tickets and one that never left your .env. It also means a password containing a comma now works, which -EnvVars cannot express.
• A preflight runs in-cluster. One VU of smoke-test.js (or of journey-test.js for the journey type, which authenticates itself) as its own Job, and the run stops with exit code 2 if not one request succeeded. The log is kept as results/<run>/preflight.log. Skip it with -SkipPreflight.
• The image tag is unique per build (<app>:local-<utc time>), because a reused tag plus imagePullPolicy: IfNotPresent is how a node serves the previous build with no rollout to notice.

When it cannot find a publish folder it asks for a Dockerfile instead, and the runner warns by name when a COPY source is not in the build context - Docker's own error for that ("failed to compute cache key") never says the context is what is wrong.

The wizard drives scripts/deploy-and-test.ps1 and prints the equivalent plain command before it does anything, so nothing here is a feature you cannot run yourself in CI.

Testing an endpoint that needs authorization

A protected route answers 401, which k6 records as a failed request: an unauthenticated run is a fast, 100%-failed test that measures nothing - and the preflight will refuse to start a longer profile when not one request succeeded, which is the intended behaviour rather than a broken build. Configure one of two mechanisms.

Let the scenario log in - the mode for anything long. lib/auth.js fetches a token, caches it per VU and renews it before expiry:

# k6\.env
AUTH_MODE=json-login
LOGIN_URL=https://idp.example.com/accounts/login
JOURNEY_USERNAME=someone@example.com
JOURNEY_PASSWORD=...
AUTH_TOKEN_PATH=data.accessToken        # where the token sits in the response

pwsh -File scripts/run-cluster-test.ps1 -SkipDeploy -TestType load `
  -TargetUrl http://api-service:8080/api/user/bank-accounts

The wizard puts the password into a Kubernetes Secret, so it never reaches results/<run>/k6-job.yaml. For an OAuth2/OIDC token endpoint instead, use AUTH_MODE=oauth2 with AUTH_TOKEN_URL/AUTH_CLIENT_ID/AUTH_CLIENT_SECRET.

Or paste a token - fine for minutes, not for a soak, because it is read once when the scenario loads and cannot be renewed:

REQUEST_HEADERS=Authorization: Bearer eyJ...

Two things break this inside a cluster, and both are worth checking before blaming the API:

• The identity provider cannot be resolved. Cluster DNS often cannot resolve external names at all, and the login then fails with lookup <host> on <ip>:53: server misbehaving before a single request is sent - so the run reports a DNS error as a test failure. -ResolveHost <host> resolves the name on the machine running the runner, where DNS does work, and pins it into the Job's /etc/hosts; what was pinned is recorded in meta.json. scripts/run-cluster-test.ps1 does this automatically for the target and for LOGIN_URL, and never for a cluster-local name - pinning api-service would send every request of a multi-replica run to one pod.
• The cluster cannot reach the identity provider at all. Pinning fixes name resolution, not routing: if there is no egress, the login still fails, and it will fail the same way from any other pod in the cluster.

If requests come back 401 with a token attached, the token is being rejected rather than missing. That is a fact about the deployment - its issuer, audience and signing key come from its own configuration - and not about the load test. Confirm by running the same scenario against an environment where those credentials are known to work.

Many users, not many sessions of one user

Thirty VUs sharing one account is not a test of the product: they serialise on that account's rows in the database, share its cache, and share its rate limit. The result can be a knee that belongs to the test rather than to the system. So the kit takes a pool of accounts, one per virtual user.

// k6/users.json  - gitignored; k6/users.example.json is the committed template
[
  { "username": "loadtest01@example.com", "password": "..." },
  { "username": "loadtest02@example.com", "password": "..." },
  "loadtest03@example.com:password-in-a-string-is-fine-too"
]

# Local: k6\users.json is picked up automatically, so this is the whole command.
pwsh -File k6/run-journey.ps1 -Vus 30 -Duration 5m

# Cluster: the pool goes into a Secret, so it never reaches results/<run>/k6-job.yaml.
pwsh -File scripts/run-cluster-test.ps1 -SkipDeploy -TestType journey `
  -TargetUrl http://api-service:8080 -EnvVars TARGET_VUS=30,TEST_DURATION=5m

• Every scenario supports it, not just the journey: rate, load, stress, soak and spike authenticate through lib/auth.js, which picks the account per VU.
• The journey logs in per VU rather than once in setup(), because setup() runs in its own context where there is no VU to attribute an account to. Every request is then tagged account=user-01, user-02, ... so an outlier account is visible instead of averaged into one latency number. The tag is an index, never the address: tags end up on dashboards.
• A VU keeps its account for the whole run. That is load-bearing, not cosmetic - the token cache is per VU, so a VU that switched account would keep sending the previous account's token.
• Accounts are never guessed at. Duplicate entries, a missing password, a malformed string entry and an empty file are all rejected before the run, with every problem listed at once and the account named. Malformed pools are checked twice on purpose: the runners catch a bad file early, and k6/scripts/lib/users.js owns the per-entry rules.
• Fewer accounts than VUs is reported, not silently shared: the runner says so, and the run log names the account each VU took.
• Account names appear once per VU in k6.log (an account name is not a credential, and "VU 7 is the outlier" is unanswerable without it). Use dedicated test accounts rather than a shared privileged one.
• fetchTokenForSetup() refuses to run with a pool: it fetches one token for the whole run, so every VU would authenticate as the same account while appearing to support many.

Two things worth knowing about the login burst, both measured on a real run: each VU logs in on its first iteration, so 30 VUs means 30 logins at once - if your identity provider rate-limits its token endpoint, that is where it shows. And if AUTH_MODE is left on for a journey run, lib/auth.js logs in as well, doubling the burst; the runners pass AUTH_MODE=off, and the scenario warns when something else did not.

Running the journey as a stress test

The journey answers which requests, not how much: the profile is a separate choice, and the journey takes all four of the kit's load shapes. (The single-endpoint scenarios have one shape each - stress-test.js is stepped, load-test.js ramps, spike-test.js spikes.)

# Local
pwsh -File k6/run-journey.ps1 -Profile stress -EnvVars STEP_VUS=10,STRESS_STEPS=4,STEP_DURATION=1m

# Cluster
pwsh -File scripts/run-cluster-test.ps1 -SkipDeploy -TestType journey -JourneyProfile stress `
  -TargetUrl http://api-service:8080 `
  -EnvVars STEP_VUS=10,STRESS_STEPS=4,STEP_DURATION=1m

JOURNEY_PROFILE	shape	knobs
steady (default)	TARGET_VUS for TEST_DURATION, constant	TARGET_VUS, TEST_DURATION
load	ramp up, hold, ramp down	TARGET_VUS, RAMP_DURATION, HOLD_DURATION
stress	plateaus, each STEP_VUS larger, then a plateau at 0	STEP_VUS, STRESS_STEPS, STEP_DURATION
spike	baseline, spike, back to baseline	BASELINE_VUS, SPIKE_VUS, BASELINE_DURATION, SPIKE_DURATION

Two things change with a staged profile, and both are deliberate:

• The per-step thresholds are expected to fail. That is the finding, not a broken build. The failure gate is loosened for stress and spike and does not abort - aborting would end the run exactly where the interesting part starts. A steady run still aborts on the first failed requests, because a broken login should stop in seconds.
• The run log names the shape it applied ([journey] load profile: stress - 4 plateau(s) of 10 VUs added each 1m (peak 40 VUs), then a plateau at 0), because a finished run should say what produced its numbers.

With a credential pool, a staged profile starts VUs in waves, so those logins spread across the stages instead of arriving as one burst. The wizard compares the pool against the peak VU count for the chosen profile, not against TARGET_VUS, and says so when accounts will be shared:

WARNING: The run reaches 40 VUs but the pool holds 5 account(s), so accounts will be shared

Adding a step to the journey

One line, in the STEPS table at the top of k6/scripts/journey-test.js:

const STEPS = [
  { name: "kyc-step",        path: envString("KYC_STEP_PATH", "/kyc/step"),              p95: 2000 },
  { name: "wallets",         path: envString("WALLETS_PATH", "/wallets"),                p95: 2000 },
  { name: "requests-active", path: envString("REQUESTS_ACTIVE_PATH", "/requests/active"), p95: 2000 },
  { name: "statements",      path: envString("STATEMENTS_PATH", "/statements"),          p95: 2500 },  // <- new
];

The request, its per-step metric, its threshold and its row in the console summary are all generated from that entry - there is nowhere else to edit, and a step cannot be added without being named and measured. Steps run in table order, each with its own name tag, so the summary says /wallets is the slow step while the others are fine.

Paths are relative to the base URL you configure, and the base may carry the API prefix: with TARGET_URL=https://tc-my-api.nt-development.dev/api/new-core the steps above resolve to …/api/new-core/kyc/step and so on. Each path stays overridable on its own with its <STEP>_PATH variable, which is what you want when the same journey has to run against an environment that mounts the API somewhere else.

Challenging a server, and seeing it in the server's resources

Two things decide whether a run actually loads a server, and neither is the number of VUs.

1. Use the open model - it is the only shape that can push a struggling server. In every VU-based profile (steady, load, stress, spike) each VU waits for its own response before sending the next request, so offered load falls when the target slows down: the test backs off exactly when you want to see it buckle. rate and rate-ramp offer a FIXED number of journey iterations per second instead, so a target that cannot keep up queues and its latency rises:

pwsh -File k6/run-journey.ps1 -Profile rate-ramp `
  -EnvVars "TARGET_RPS=100,RATE_STEPS=5,STEP_DURATION=1m,PREALLOCATED_VUS=50,MAX_VUS=800"

2. k6 cannot show you the server's CPU - only the server can. The run gives you achieved throughput, per-step latency, and dropped_iterations: iterations the generator could not send, now printed in the summary as a warning. If that is not zero, the generator was the ceiling and the server was never challenged - raising MAX_VUS is the test of which. For the server's own resources, pair the run's timestamps with the server's telemetry:

• the application is instrumented with Elastic APM (ElasticApmServiceName appears in its own logs), which shows requests per minute and per-transaction latency - that is the evidence the work reached the app rather than stopping at a proxy;
• CPU and memory come from wherever the app runs: kubectl top pod -n <ns> (and container_cpu_cfs_throttled_periods_total) if it is in Kubernetes, or the host's own monitoring otherwise. Watch them during the ramp, not after.

Four things that make "challenge the server" harder than it looks, all of them real here:

• A CDN in front absorbs the load. tc-my-api.nt-development.dev resolves to Cloudflare addresses, so some of what you challenge is the edge: cached responses never reach the origin, WAF and rate limits may answer for it, and a sustained high rate is a good way to get the load generator's IP blocked. Check whether the endpoint you chose is cacheable, and be ready to say which part of the path you measured.
• The generator has a ceiling of its own, and it is usually the laptop's uplink or CPU before it is k6. For a remote target, run the generator from a machine with a short, idle network path - and prefer the compose path (k6/run-journey.ps1), which uses your current k6/scripts directly, over the cluster generator, whose image has to be rebuilt to pick up changes.
• A high offered rate costs one login per VU. VU count is roughly the offered rate times how long one journey takes: 100 iterations/s with a 1.5 s journey needs ~150 VUs, so ~150 logins, and an identity provider that rate-limits the token endpoint will fail first. Either size the credential pool to that number or use a static token (REQUEST_HEADERS=Authorization: Bearer ..., with AUTH_MODE=off).
• It is a shared environment. Announce it, ramp rather than slam, keep think time non-zero for anything that writes, and stop when the numbers stop being interpretable (429s and 5xx are the point where you have found the limit, not where you should keep going).

Golden rules

• The measured configuration is the experiment. k8s/base/api-deployment.yaml runs Production, the image must be a Release build, and the CPU/memory requests and limits are part of the result. Change any of them and earlier runs are no longer comparable - record it as a new baseline. The runner now records this for you: meta.json carries both what the run asked for (api_resources_requested) and what the cluster was actually running (api_resources_observed, including the QoS class and the nodes involved), and it warns when the two disagree - which is the normal outcome of -TestOnly against a deployment somebody else sized.
• Test a real endpoint. A Swagger page is static HTML served by middleware that is normally disabled outside Development. The runner warns if your -TargetUrl contains swagger; that warning is the ghost of a bug (see P0 item 1).
• Isolate the generator. The k6 pod prefers a node without the API on it and requests 1 CPU with no CPU limit. On a single-node cluster that anti-affinity is ignored and the two compete for CPU - one more reason local numbers are only for script validity.
• Keep the evidence. Never overwrite results/; the runner creates a new directory per run and results/latest.txt points at the newest.
• Stress and spike tests are supposed to fail their thresholds. k6 exits 99 when a threshold is crossed. For those two types, a non-zero exit is the finding, not a broken build - which is why they are run by hand, not on every commit.

Which test, and where

Test	Question it answers	Profile	Home environment	Cadence
smoke	Does the script and target work at all?	1-2 VUs, <1 min	laptop, PR pipeline, production synthetic monitor	every change
load	Do we still meet the SLO at expected peak?	ramp, hold at peak, ramp down	staging (prod-shaped)	per release
stress	Where is the knee, and does it degrade gracefully?	stepped plateaus to and past failure	staging only	per major release
spike	Can it absorb a flash crowd, and how fast does it recover?	baseline -> sharp spike -> baseline	staging, before known traffic events	occasional
soak	Does anything leak, fill up, or drift over hours?	2-24h at 60-80% of peak	dedicated staging, overnight	nightly/weekly

Local, staging, or production?

• Your laptop - write and debug scenarios here. Never take capacity numbers from it: the generator and the target share one machine, the Docker VM adds a hop, and a laptop holds neither a steady clock nor a steady thermal budget.
• CI (ephemeral kind/minikube) - smoke on every pull request, plus optionally a short load run with tight thresholds. Cheap, isolated, deterministic. Keep it under about five minutes or people will route around it.
• A dedicated, prod-shaped staging cluster - the default home for load, stress, spike and soak. Same instance sizes, same replica count, same configuration and production-sized data; nothing else running on it; reset between runs. Data volume is the single biggest realism factor for a database-backed API.
• Production - only low-rate synthetic monitoring of a real endpoint, or (rarely, with approval) a read-only canary load with a slow ramp, SLO-based auto-abort, a change freeze and on-call informed. Never stress-to-destruction, never spike-to-failure, never a soak, and never load from production against shared databases, third parties or payment gateways.

Production is the most realistic environment and the least destructible one; that trade-off is exactly why prod-shaped staging exists.

Scenario knobs

Pass with -EnvVars NAME=value,NAME=value. TARGET_URL cannot be set this way: use -TargetUrl, which is validated and recorded.

Knob	Default	Applies to
TARGET_URL	(required, no default)	all
REQUEST_PAUSE	0.5 / 0.3 / 0.2 s	all (think time between iterations)
TARGET_VUS	1 / 50 / 20	smoke / load / soak
TEST_DURATION	1m	smoke
RAMP_DURATION	1m / 2m	load / soak
HOLD_DURATION	10m / 2h	load / soak
STEP_VUS	50	stress (VUs added per plateau)
STRESS_STEPS	4	stress (peak = STEP_VUS x steps)
STEP_DURATION	2m	stress
BASELINE_VUS	10	spike
SPIKE_VUS	100	spike
BASELINE_DURATION	30s	spike (also the recovery window)
SPIKE_DURATION	1m	spike

Thresholds live in the scenario files, next to the profile they belong to.

Results, baselines and gating

Every run gets results/<utc timestamp>-<type>/:

File	What it is
k6.log	the full container log, including the human-readable summary
summary.json	machine-readable summary, schema perftest.summary/v1
meta.json	target URL, image, git revision, env knobs, pod, k6 exit code, the generator's CPU request, and the measured configuration (api_resources_requested / api_resources_observed)
k6-job.yaml	the exact Job manifest that was applied (cluster runs)
preflight-job.yaml	the Job manifest of the 1-VU preflight, when one ran
preflight.log	the preflight's own k6 log
runtime-settings.yaml	the exact Deployment patch this run applied: image, replicas, environment and resources. Written when the run deployed the API; absent under -TestOnly, where api_resources_observed in meta.json describes what was reused instead

The summary travels through the log between two markers, which is why it works identically for compose and for a Job - no shared volume, no sidecar, no object storage, and nothing to collect from a container that has already exited.

scripts/compare-summary.ps1 compares a run against baselines/<type>.json and exits non-zero when p95/p99/avg latency, TTFB, error rate or throughput regresses beyond -TolerancePercent (default 10%). Improvements always pass. Recording a baseline from a run whose thresholds failed is refused.

Do not set the tolerance to 1% and expect a stable gate: run-to-run variance on real infrastructure is bigger than that, and a gate that cries wolf gets ignored.

Did the generator limit the result?

This is the first question to ask about any surprising number. The runner gives you some of it, but the check is manual:

1. CPU: kubectl top pod -n perf-test -l job-name=k6-test-load. If the k6 pod is pinned near its request while the API pod is idle, suspect the generator. (Needs metrics-server.)
2. Node placement: kubectl get pod -o wide - if k6 and the API are on the same node in a single-node cluster, you are measuring contention. The runner now does this check for you and records k6_node, api_nodes and k6_colocated_with_api in meta.json; it warns on the console when the two pods landed together.
3. Sanity run: point the same scenario at a trivially fast endpoint in the same cluster and compare the achieved http_reqs.rate. If it is close to your real run's RPS, you found the generator's ceiling, not the API's.
4. Little's Law: VUs ~= RPS x latency. If the numbers disagree by an order of magnitude, one of the three is not what you think it is.

Verification status

Two flags exist specifically so the runner can be checked without a cluster: -EmitJobManifest (write the Job manifest and exit) and -FromLog (re-parse an already-collected log, which exercises summary extraction, summary.json and meta.json).

node k6/tests/summary.test.mjs                       # summary library, all assertions
kubectl kustomize k8s/overlays/local                 # base + overlay render cleanly
pwsh -File scripts/deploy-and-test.ps1 -TargetUrl http://api-service:8080/api/orders -TestType load -EmitJobManifest ./job.yaml
pwsh -File scripts/deploy-and-test.ps1 -TargetUrl http://api-service:8080/api/health -TestType smoke -FromLog results/<run>/k6.log

Verified against a live four-node-path minikube cluster (three nodes, a real .NET API deployment, the in-cluster Prometheus and Grafana), as in-cluster Jobs:

• a smoke run and a stress run, -TestOnly, exit code 0, artefacts complete (k6.log, summary.json, meta.json, k6-job.yaml);
• meta.json recording the configuration read back from the running Deployment: 2 CPU / 2Gi request and limit, qos_class: Guaranteed, replicas_ready 1/1, with api_nodes: [minikube-m02] against k6_node: minikube-m03 and k6_colocated_with_api: false - the Job's anti-affinity honoured on a multi-node cluster, which is the check the README used to describe as a manual step;
• the run's testid reaching both the applied Job manifest and Prometheus, queried through the API server: sum(k6_http_reqs_total{testid="..."}) returned the run's own request count (564,884), and the testid label carried both the run and its <testid>-preflight sibling - so the dashboard's run selector filters a cluster run the same way it filters a compose run;
• the preflight gate: results/<run>/preflight.log and preflight-job.yaml written, preflight: "passed" and preflight_failure_rate: 0.0 recorded in meta.json.

That last point found a real defect in this change set: extracting the Job runner dropped -PrometheusWriteUrl at the call site, so the pod received no K6_OUT, pushed nothing, and the run still reported success with an empty dashboard - the exact failure mode the compose README documents. Caught only by querying Prometheus for the series rather than trusting the exit code.

Verified (no cluster, no Docker daemon required): the summary library against synthetic k6 payloads; the result path end to end, including exit code 99 on a crossed threshold; test-type validation; the baseline gate on pass / regression / missing-metric / failing-baseline cases; kubectl kustomize for the base, the checked-in overlay and a generated per-run overlay (including digest pinning, namespace override, replicas and environment); the generated Job manifest parses as a Kubernetes object.

Also verified, by extracting the runner's own function definitions from its AST and calling them against canned kubectl output: the Deployment readback for a Guaranteed spec, a Burstable one, one with no resources block at all (reported Burstable rather than a false Guaranteed), a spec whose readyReplicas is below replicas, a failed kubectl and empty output (both yield "unavailable" instead of costing a run its results); the node-placement lookup, including duplicate nodes and failure; and the api_resources_requested / api_resources_observed nesting surviving the ConvertTo-Json -Depth 8 round trip that meta.json is written with.

The Dockerfile context check was run against a hand-written Dockerfile exercising each shape it has to ignore (--from=, an ADD URL, a glob, a multi-line COPY) and each one it must catch, plus the kit's own k8s/app/Dockerfile against a real publish folder, and an unreadable path (no throw, no warnings). The generated overlay is now also rendered through kubectl kustomize from the runner's own function, asserting the image it claims to set - the check that would have caught the placeholder-image defect before it reached a cluster. The pod diagnosis was run against a live stalled rollout, where it named ImagePullBackOff on your-api:latest in one line. The wizard's -DryRun was run for: the journey type against the real k6\.env, a load run with a 4 GiB limit, an explicit Dockerfile with -BuildContext and -BuildArg, -SkipDeploy, -SkipBuild without -Image, -SkipObservability, a missing -TestType under -NonInteractive, a production-looking host, and a missing Dockerfile - and the secret's values were asserted absent from the output while only its key names were listed.

Not verified here - no Docker daemon was reachable from the environment where these changes were made (its API is a named pipe, which the sandbox refuses), and kubectl was pointed at an existing cluster that already had a Deployment and the generator image loaded. So the following have been written and dry-run, never executed:

• the deploy path (-not -TestOnly): docker build, minikube image load, kubectl apply -k, the rollout, and therefore -MemoryLimit 4Gi actually being applied to a pod;
• -CredentialsSecret, envFrom in a live pod, and the wizard's Secret creation (deliberately not run: it would have written the real account password into the cluster to test something the manifest shape already covers);
• scripts/run-cluster-test.ps1 end to end - its -DryRun plan, parameter handling and error paths were exercised, but not its first real invocation;
• the Grafana link itself (the data behind it was queried directly instead), and the minikube service fallback for a NodePort that is not reachable on 127.0.0.1 - which is the case on this machine;
• a run that crosses its thresholds in-cluster (exit code 99).

Verify in this order:

docker build -f k6/Dockerfile -t k6-custom:local .     # generator image builds
minikube image load k6-custom:local                    # and the cluster can see it
kubectl kustomize k8s/overlays/local | Select-String 'image:'   # must print service-api:local
kubectl apply -k k8s/overlays/local --dry-run=server   # cluster-side validation
# A route that actually exists on YOUR app: a 404 is a failed request, and k6
# reports it as a fast, healthy-looking run.
pwsh -File scripts/deploy-and-test.ps1 -TargetUrl http://api-service:8080/<a real route> -TestType smoke
pwsh -File scripts/run-cluster-test.ps1 -TestType smoke -DryRun   # the wizard's plan
pwsh -File scripts/run-cluster-test.ps1 -TestType smoke           # then the real thing

Troubleshooting

Symptom	Likely cause and fix
k6 pod Pending, "cannot be scheduled"	Not enough CPU for the request. Lower -K6CpuRequest, or free capacity.
k6 pod ImagePullBackOff	The image is not visible to the cluster: kind load docker-image / minikube image load / push to a registry. The generator image is never pulled from a registry, so it has to have been loaded - and the runner now warns before applying the Job when it could not load it, naming the daemon when that is the reason. -TestOnly loads it too; it used to skip that step, which is what leaves a -TestOnly run against kind/minikube stuck here with only a k6-job.yaml in results/ to show for it.
k6 fails with The moduleSpecifier "/scripts/x.js" couldn't be found on local disk	The generator image is older than your workspace: it was loaded into the cluster before the scenario you are running existed. The file is plainly in k6/scripts and plainly inside the image on the host, which is what makes this confusing. Rebuild and reload (docker build -f k6/Dockerfile -t k6-custom:2 . then minikube image load k6-custom:2), and prefer a new tag: an untouched tag can be served from a node's cache, so the rebuild alone may not be enough.
"k6 output contained no summary block"	The container failed before the end-of-test summary. Read k6.log or kubectl logs -n perf-test -l job-name=....
Every response is 404	Wrong path, or the wrong port (the Service is 8080 end to end), or the app runs with an environment where the route is not registered.
Latency is flat and suspiciously good	You are probably not load-testing the app: check what the endpoint actually does.
rollout status times out: "timed out waiting for the condition"	The new pod never became Ready, and kubectl does not say why. The runner now prints the reason per pod before failing (ImagePullBackOff, a probe that never passed, a container that exits immediately, a pod stuck terminating). If the waiting reason is ImagePullBackOff on your-api:latest, the overlay did not substitute the image at all: check it with kubectl kustomize k8s/.generated/<namespace> | Select-String 'image:'. That was a real defect in this kit, fixed by setting the image in the Deployment patch rather than through kustomize's images: transformer.
A rollout leaves two pods Running	Normal while a rollout finishes: the old pod is terminated only after the new one is Ready. It is only a problem when it never finishes.
k6 log: lookup <host> on <ip>:53: server misbehaving	The cluster's DNS could not resolve an external name, so the request never left the pod (measured here: CoreDNS timing out against Docker Desktop's resolver while the host resolved the same name fine). Pass -ResolveHost <host>, which resolves it where DNS works and pins it into the Job's /etc/hosts; scripts/run-cluster-test.ps1 does this automatically for the target and for LOGIN_URL.
Every request 401	The route needs a token. See "Testing an endpoint that needs authorization" above - and remember a 401 counts as a failed request, so it shows up as http_req_failed: 100%.
Grafana link does not load	A NodePort is not published on your host's loopback on minikube or kind, so the link needs a kubectl port-forward to exist first. The wizard starts one itself (supervised, so it survives pod replacements). If a link that was working stops, the forward exited - pwsh -File scripts/expose-observability.ps1 keeps it alive and reconnects; see "When a working link stops working" below.
Exit code 99	Thresholds crossed - a real finding. Expected for stress and spike.
Local run cannot resolve the target	The runner now rejects a localhost/127.0.0.1 target instead of warning: inside the container that is the container itself, and every request fails in ~2ms. Use host.docker.internal, or -AllowLocalhostTarget if you truly mean the k6 container.
Local run works but Grafana shows no data	In order of likelihood: (1) the time range no longer covers the run - k6 writes samples only while the test runs, so a finished run is invisible to an instant query at "now"; use the Grafana link the runner prints, or widen the range; (2) k6 has no output enabled (K6_OUT, not just K6_PROMETHEUS_RW_SERVER_URL - the URL alone sends nothing); (3) Grafana has more than one datasource, or none (a hand-added Prometheus with an empty URL breaks every panel); (4) Prometheus rejects remote writes (needs --web.enable-remote-write-receiver, plus --enable-feature=native-histograms for the histogram pushes). Check http://127.0.0.1:9090/graph first - see k6/README.md.
http://localhost:3000 hangs	Docker Desktop publishes these ports on IPv6 too, and its IPv6 proxy does not answer on this machine: 127.0.0.1 works, localhost/[::1] time out. Every documented URL now uses 127.0.0.1.

What changed (P0: making the numbers trustworthy)

1. Targeted a real endpoint. The old default was http://api-service:8080/swagger/index.html - a static Swagger page, on a container running ASPNETCORE_ENVIRONMENT=Production where Swagger is normally not registered. -TargetUrl is now mandatory and validated.
2. Stopped measuring a Debug build. The build arg was ENVIRONMENT=Development while the container ran as Production. -BuildEnv now defaults to Release and warns loudly on Development.
3. Fail fast instead of hanging. kubectl wait --for=condition=complete never returns early when k6 exits 99, so a failing test blocked for the full 30 minute timeout. The runner now waits for complete or failed, reports k6's own exit code, and short-circuits on ImagePullBackOff and unschedulable pods.
4. Replaced the kubectl patch hack with kustomize. Strategic-merge patches on containers and Service ports add entries rather than replacing them: the Service ended up advertising a port 80 route with nothing listening behind it, which is exactly why the scripts' http://api-service/ default could never work.
5. Added readiness and liveness probes. Without them rollout status goes green the moment the container is running, so the test could start before Kestrel was listening.
6. Kept the evidence. Log, machine-readable summary, metadata and the applied Job manifest are written to results/, and compare-summary.ps1 turns that into a release gate against a recorded baseline.
7. Protected the generator. The k6 pod requests real CPU, has no CPU limit, and prefers a node without the API on it.
8. Removed the silent traps. A -TestType with no scenario file now lists the real options (this is how -TestType load used to be accepted with no load-test.js); checks has a threshold so a failing assertion fails the run; response bodies are discarded so the generator's memory is not under test; the k6 image tag is pinned.
9. Recorded the configuration the numbers came from. The runner used to archive the Job manifest but nothing about the API it was aimed at, so a result could not be compared with another result or explained afterwards - against a rule this README states twice. meta.json now carries what the run asked for, what the cluster was actually running (read back after the run, with QoS class and node placement, and a warning when the two disagree), and the applied Deployment patch is copied into results/<run>/runtime-settings.yaml instead of being overwritten by the next run's overlay. The console summary also prints iteration_duration, the one metric that answers "how long does the whole flow take" - which no per-request percentile can, and which the journey test exists to measure.
10. Made the cluster path usable without reading the source. -Dockerfile and -BuildContext replace the "guess the context three directories up" build, and a missing COPY source is reported by name instead of as Docker's cache-key error; scripts/run-cluster-test.ps1 prompts for what it needs, builds, deploys, runs the preflight and prints the Grafana link. The k6 Job is now tagged with a testid (the same string as the results directory), so a cluster run can be filtered in Grafana like a compose run; credentials can be injected from a Secret via -CredentialsSecret instead of being written into the archived Job manifest; and -Preflight refuses to start a long profile against a target that never answered, in-cluster, for the same reason the compose runner has always done it. -TestOnly also no longer skips loading the generator image into kind/minikube, and says so when the Docker daemon is what stopped it - the failure that leaves a Job in ImagePullBackOff with a perfectly good k6-job.yaml and nothing else.
11. Stopped deploying the placeholder image. The overlay set the API image with kustomize's images: transformer, naming your-aspnet-api - a container no manifest has. The transformer matches on the image name, so it matched nothing, substituted nothing, and every deploy wrote k8s/base's placeholder your-api:latest into the Deployment. The symptom was a pod in ImagePullBackOff and a rollout timeout 300 seconds later that never mentioned the image; the damage was worse than the symptom, because the apply replaced whatever image was working, and only maxUnavailable: 0 kept the old pod alive to serve traffic. The image is now set by the same Deployment patch that sets the resources and the environment, the runner reads it back immediately after the apply and refuses to wait for a rollout that cannot succeed, and a failed rollout prints why each pod is not ready instead of kubectl's bare "timed out waiting for the condition".
12. Made authenticated cluster runs possible at all. They were not: cluster DNS could not resolve the identity provider, so every scenario that logs in died before its first request with lookup <host> on 10.96.0.10:53: server misbehaving while the same name resolved perfectly from the host. -ResolveHost resolves a name on the machine running the runner and pins it into the k6 Job's /etc/hosts, and records the mapping in meta.json; the wizard does it for the target and LOGIN_URL automatically, and skips cluster-local names. Measured on the same run before and after: one request, 0.00 ms average and a DNS error, then twelve requests at 222 ms against the identity provider. The README also gained the recipe for a protected endpoint, since a 401 is a failed request and looks like a broken API.
13. Stopped printing a Grafana URL that only works on one kind of cluster. The in-cluster dashboard is a NodePort, which is opened on the node network - on minikube and kind nothing is published on 127.0.0.1, and the node addresses are not routable either (all three verified unreachable here while Grafana itself was healthy). Both the observability script and the wizard now probe the port, pick a free local port for the port-forward, and print a URL that answers. The free-port part matters more than it sounds: 3000 is the compose stack's own Grafana, and forwarding onto it shows the wrong dashboard against the wrong Prometheus.
14. Stopped replacing the Grafana pod on every run. A port-forward is attached to the pod behind the Service, so replacing that pod breaks every dashboard route - and the observability deploy restarted Grafana unconditionally on each invocation, which on this cluster meant nine ReplicaSets in a day and a dead link every time. It now restarts only when the dashboards or provisioning actually changed, proved by comparing the ConfigMaps' resourceVersion across the apply. The wizard also starts the forward itself rather than printing a URL that needs one, and says how to get the link back when a pod replacement kills it.
15. Added a pool of test accounts, one per virtual user. Thirty VUs sharing one account serialise on that account's rows, share its cache and share its quota, so the run can report a knee that belongs to the test. k6/users.json (gitignored) now supplies one account per VU for the journey and for every other authenticated scenario, each request is tagged account=user-01, and the pool travels to a cluster run inside the credentials Secret - verified: it appears nowhere in the archived Job manifest. The scenario logic is covered by k6/tests/journey-pool.test.mjs, which runs the real journey with k6's modules stubbed and asserts that each VU logs in as its OWN account - the failure this guards against is silent, because thirty VUs authenticating as one account looks like a perfectly healthy run.
16. Made the dashboard link survive the thing that keeps killing it. A kubectl port-forward is attached to the pod behind the Service, so replacing that pod ends it - and on this cluster a long-lived stream was also torn down by the host (wsasend: An existing connection was forcibly closed) with the same symptom, two hours after the last pod replacement. Either way the browser reports a connection error for a Grafana that is healthy. scripts/expose-observability.ps1 now supervises the forward: fixed local port, reconnect on exit, and it says why the previous one ended. Verified by deleting the Grafana pod under a running forward - two reconnect attempts while the new container started, then the original URL served the dashboard again, untouched. The wizard starts the supervisor rather than a bare kubectl.

Not done yet (P1/P2, in the order I would take them)

1. Open-model load. load-test.js still uses VU-based stages, which is a closed model: offered load falls when the API slows down, so the test backs off exactly when you want to see the system buckle. Move load/stress/spike to constant-arrival-rate / ramping-arrival-rate.
2. A real user journey. One GET is not a workload. Add authentication, a read/write mix, realistic payloads, and setup()/teardown() for test data.
3. Observability for the system under test. k6 alone shows latency, not cause. Add kube-state-metrics, node-exporter and .NET runtime metrics (OpenTelemetry or dotnet-counters), then put node/pod CPU and memory on the same dashboard as the k6 latency panel.
4. A metrics sink for cluster runs. Cluster Jobs report via artefacts and logs only. The local compose stack now sends k6 metrics to Prometheus by remote write (k6/docker-compose.yml, k6/grafana/); point cluster runs at a Prometheus you keep - the same remote-write output works from inside a cluster, it just needs a reachable URL and K6_OUT set on the Job.
5. Decouple from the application repository. Consume a pinned image digest and delete the -ApiPath build path, the ..\..\.. context guess and the nuget.config temp-Dockerfile rewrite.
6. CI. Smoke on every pull request on ephemeral kind; nightly load and soak in staging; stress and spike by hand with approval; publish artefacts and trends.
7. k6-operator and distributed load. Replace the hand-rolled Job with a TestRun resource when a single generator caps out, and add chaos/HPA-validation scenarios.
8. Decide the fate of the two InfluxDB-era dashboards in k6/dashboards/. Done. They were InfluxQL against a k6influxdb datasource and could not work against Prometheus, so they and the orphaned k6-influxdb-1 container are gone.
9. Align the generator version. Done, and the reason given here was wrong: experimental-prometheus-rw is accepted by both k6 0.54.0 and 1.3.0 (verified by asking each image to list its output types), so the pin could be raised without touching the output name. Both files now pin 1.3.0.

Grafana, or a result file?

Both. They answer different questions, and neither replaces the other:

	Grafana + Prometheus (live)	results/<run>/ (durable)
Answers	"what is happening now", and how this run compares to the last five	"exactly what did this run measure"
Lifetime	7-day retention, then gone	permanent, diffable, reviewable
Needed for	watching a run, finding the knee, spotting soak drift, overlaying runs	CI gates, audits, regression detection, post-mortems
Cannot do	gate a pull request; survive a volume wipe	show the shape of a two-hour soak

A dashboard cannot gate a pull request and a JSON file cannot show you a rising tail at minute 90. The kit keeps both: every run writes summary.json/meta.json, and the same run streams into Prometheus and appears on the dashboard.

Reaching the in-cluster Grafana

/d/k6-load-test is served by a NodePort, and a NodePort is opened on the node's network - not on your machine's loopback. Measured on the minikube this kit was verified against: 127.0.0.1:30030 refused the connection, all three node addresses (192.168.49.2/.3/.4:30030) were unreachable from the host, and Grafana itself was perfectly healthy and serving the dashboard. Docker Desktop's built-in Kubernetes is the case where 127.0.0.1:30030 does work, which is where that URL came from - it was printed unconditionally, so on minikube it was a dead link and a working Grafana looked broken.

kubectl port-forward -n perf-observability svc/grafana 3001:3000   # then http://127.0.0.1:3001/d/k6-load-test
kubectl port-forward -n perf-observability svc/prometheus 9091:9090
minikube service -n perf-observability grafana                     # or let minikube tunnel it for you

Use a free local port, and specifically not 3000: that is where the compose stack publishes its own Grafana, so forwarding the in-cluster one onto it either fails with "address already in use" or shows a dashboard querying a different Prometheus, which reports "No data" for a cluster run.

scripts/run-cluster-test.ps1 no longer leaves this to you. If the NodePort does not answer, it probes for a free port above 3000, starts the port-forward itself, waits until the port answers, and only then prints a link - announcing the pid so you can stop it, and reusing a forward an earlier run left running instead of stacking another. If no route can be established it prints no link and says why, because a URL that refuses to connect is indistinguishable from a broken Grafana:

Grafana (this run, already zoomed to its window):
  http://127.0.0.1:3001/d/k6-load-test?from=...&to=...&var-testid=...&var_test_type=...
  (a NodePort is not reachable on 127.0.0.1 on minikube or kind, so a port-forward was started
   and left running: pid 29884, stop it with Stop-Process -Id 29884)

To check the metrics independently of the dashboard (no port-forward needed), query Prometheus through the API server:

kubectl get --raw '/api/v1/namespaces/perf-observability/services/prometheus:9090/proxy/api/v1/query?query=sum(k6_http_reqs_total{testid="<testid>"})'

One thing to know when querying latency by hand: k6 pushes native histograms, so the PromQL is histogram_quantile(0.95, sum(rate(k6_http_req_duration_seconds{testid="..."}[1m])))

• no _bucket suffix and no by (le). The classic form returns "no series" and looks like missing data.

When a working link stops working

A kubectl port-forward is attached to the pod behind the Service, and it is fragile in three separate ways, all of them observed on the cluster this kit was verified against:

• replacing that pod ends it with error: lost connection to pod - and replacing the pod is what any dashboard change does;
• the stream to the API server can drop on its own. On Docker Desktop a long-lived connection is torn down with wsasend: An existing connection was forcibly closed by the remote host, and kubectl reports it as lost connection to pod even though the pod has been up for two hours;
• the process can simply be closed, or die with the terminal that started it.

In all three cases the browser shows a connection error for a Grafana that is perfectly healthy, which sends you to debug the wrong thing. So use the supervisor, which keeps the forward running on a fixed port and reconnects whenever kubectl exits:

pwsh -File scripts/expose-observability.ps1                    # grafana, first free port from 3001
pwsh -File scripts/expose-observability.ps1 -Service prometheus -LocalPort 9091

It prints the URL, says why a forward ended, reconnects every 2 seconds, and keeps the same URL across pod replacements - verified by deleting the Grafana pod underneath a running forward: the replacement took two reconnect attempts while the new container started, and the original link then served the dashboard again with no intervention. Ctrl+C stops it. scripts/run-cluster-test.ps1 starts this supervisor itself, so a run's own link survives its own deployment.

scripts/deploy-observability.ps1 also used to restart Grafana unconditionally on every invocation, so on this cluster there was a new Grafana pod on nearly every test run (nine ReplicaSets in one day, all but one scaled to zero) and any forward or open dashboard died with it. It now compares the ConfigMaps' resourceVersion across the apply - kubectl apply of identical content is a no-op, which makes it an exact "nothing changed" signal - and restarts only when the dashboards or provisioning really changed.

Local observability: what was broken

The Prometheus/Grafana path was reported as "works with InfluxDB, not with Prometheus". It was investigated on the real stack rather than by reading, and the sink was never the problem - k6's metrics were arriving, and 61 k6_* metric names were sitting in Prometheus. Five separate defects made it look dead:

1. The dashboard's time range did not cover the run - by far the most important. k6 writes samples only while a test runs. Evaluating the dashboard's own queries at a run's timestamp returned 56 metric names and 0.83 req/s; evaluating the same queries at "now", after the run, returned zero series. Every panel said "No data" and the pipeline was pronounced broken. Fixed by a wide default range and, properly, by having the runner print a link pinned to the run's exact window.
2. localhost hangs on this machine. http://127.0.0.1:9090 answers; http://localhost:9090 and http://[::1]:9090 time out, because Docker Desktop also publishes on IPv6 and its IPv6 proxy does not respond. A browser falls back to IPv4 and looks fine, so only scripts and health checks hung. All ports are now bound to 127.0.0.1, and every documented URL uses it.
3. Two datasources, one of them broken. A hand-added prometheus with an empty URL sat next to the provisioned one. The old README told you to import dashboard 19665 by hand, and that import binds to whatever datasource it likes - including this one, which renders every panel empty. The duplicate is deleted and the kit now ships its own dashboard instead of pointing at Grafana.com.
4. Cumulative latency statistics. k6's gauge trend stats are cumulative over the run: measured across 67 samples of one run, k6_http_req_duration_p95 held a single value (0.0243) and _max never moved. They converge and then stop, so a spike or a slow soak drift is invisible - the exact signal a soak exists to find. k6 now pushes native histograms, and the dashboard reads histogram_quantile(0.95, sum(rate(k6_http_req_duration_seconds[...]))), which on the same kind of run returns a real moving series (52.56, 131.79, 198.43, ...). The panels fall back to the gauges automatically if histograms are switched off.
5. test_type never reached Prometheus. A tag set in a scenario's options.tags does not arrive, while the same tag passed as a CLI --tag does. Both runners now pass it on the command line, so the dashboard's test-type filter works.

Also fixed while in here: -EnvVars A=1,B=2 was broken under pwsh -File (which does not split commas), so the README's own example fed k6 a single malformed value; and the orphaned k6-influxdb-1 container plus the two dead InfluxQL dashboards are gone, which removes the orphan warning from every run.

The kit now also ships a demo target (traefik/whoami, always up), so "is the kit broken or is my API broken?" is a one-command question.

Rules worth writing on the wall

Never load-test production to destruction. Never run a load test from a laptop and call it a capacity number. Never compare two runs whose configuration changed. Never record a baseline from a run whose thresholds failed. And always smoke first.