Skip to content

Add parquet dashboards and alerts - #68

Open
friedrichg wants to merge 2 commits into
mainfrom
parquet-converter-resources-dashboard
Open

friedrichg wants to merge 2 commits into
mainfrom
parquet-converter-resources-dashboard

Conversation

@friedrichg

@friedrichg friedrichg commented Oct 15, 2026 •

Copy link
Copy Markdown
Member

What this PR does:

Adds two dashboards and an alert group for the parquet converter and the parquet queryable.

Cortex / Parquet, in the same layout as Cortex / Compactor:

  • Converter: block conversions and failures / sec (cortex_parquet_converter_blocks_converted_total, cortex_parquet_converter_block_convert_failures_total), tenants owned per instance, the conversion delay histogram (cortex_parquet_converter_convert_block_delay_minutes), and the latest conversion duration per tenant (a gauge, top 10).
  • Bucket: the 10 tenants with the most unconverted blocks and the most parquet blocks (cortex_bucket_parquet_unconverted_blocks_count, cortex_bucket_parquet_blocks_count, reported by the compactor).
  • Querier and Ruler: blocks queried / sec and operations / sec by type (parquet, tsdb, mixed), and the parquet shard cache hit ratio and item count.
  • Object Store: the parquet-converter bucket client.

All metrics exist in Cortex v1.21.1.

Cortex / Parquet Converter Resources, with the same rows and panels as Cortex / Compactor Resources, for the parquet-converter container:

  • CPU and Memory: CPU, Memory (workingset), Memory (go heap inuse)
  • Network: Receive Bandwidth, Transmit Bandwidth
  • Disk: Disk Writes, Disk Reads, Disk Space Utilization

The parquet converter downloads whole TSDB blocks to its local disk before it converts them, so its disk and network use need the same view as the compactor.

Alerts (cortex_parquet_alerts, with a playbook entry for each):

Alert Severity Fires when
CortexParquetConverterFailingConversions warning a tenant has conversion failures and no conversions for 1h
CortexParquetConverterNotConverting critical a tenant has blocks that are due for conversion and no conversions for 3h
CortexParquetConversionDelayHigh warning more than 20% of the conversions finish more than 5h after the block upload (the top bucket of the delay histogram)
CortexParquetConverterRingUnhealthy warning the parquet-converter ring has unhealthy members for 15m
CortexParquetQueryableMostlyTSDB warning more than 70% of the blocks that the parquet queryable finds are TSDB blocks for 2h
CortexParquetCompactionStuckAtFirstRange critical more than 80% of the blocks of a tenant with more than 500 blocks are neither parquet blocks nor due for conversion, for 6h

The last alert covers a case that the parquet metrics alone cannot show. The converter only takes blocks wider than the first -compactor.block-ranges range, so when the compactor cannot compact a tenant, cortex_bucket_parquet_unconverted_blocks_count stays 0 and almost nothing is converted.

Also adds parquet_converter: 'parquet-converter.*' to job_names and instance_names in the mixin config.

Checks:

  • make lint-mixin-with-jsonnetfmt passes.
  • make build-mixin generates cortex-parquet.json and cortex-parquet-converter-resources.json.
  • make lint-playbooks passes. The alerts add no mixtool lint findings.
  • mixtool lint reports 936 findings against 838 on main. The new findings are the same target-instance-rule, target-job-rule, template-datasource-rule and template-job-rule findings that Cortex / Compactor (78) and Cortex / Compactor Resources (22) already have, because the dashboards use the same helpers.
  • Every query of both dashboards was run against a Cortex 1.21.1 cluster with the parquet converter enabled, with the job selectors changed to that deployment's job names. All return series, except the two limit lines (the converter there has no CPU or memory limits) and the failure counter (no failed conversions). The six alert expressions also run there. Only CortexParquetCompactionStuckAtFirstRange fires, for a 100M-series tenant whose compactors cannot finish the first-range merge (about 10,000 2h blocks and 2 parquet blocks).

Checklist

  • CHANGELOG.md updated

@friedrichg friedrichg changed the title Add Cortex / Parquet and Cortex / Parquet Converter Resources dashboards Add parquet dashboards and alerts Oct 16, 2026
@friedrichg
friedrichg force-pushed the parquet-converter-resources-dashboard branch from b4ffd3f to f1fa10b Compare October 16, 2026 03:48
The parquet converter downloads whole TSDB blocks to local disk and
converts them to parquet, and the queriers and rulers read the parquet
files through the parquet queryable. Add dashboards and alerts for
this pipeline.

Cortex / Parquet Converter Resources has the same rows and panels as
Cortex / Compactor Resources (CPU and memory, network, disk), for the
parquet-converter container.

Cortex / Parquet shows the pipeline end to end, in the same layout as
Cortex / Compactor:

- Converter: block conversions and failures, tenants owned, the
  conversion delay histogram and the latest conversion duration.
- Bucket: the tenants with the most unconverted blocks and the most
  parquet blocks, from the compactor's bucket index gauges.
- Querier and Ruler: blocks queried and operations by type (parquet,
  tsdb, mixed), and the parquet shard cache hit ratio and items.
- Object Store: the parquet-converter bucket client.

The cortex_parquet_alerts group, with a playbook entry for each alert:

- CortexParquetConverterFailingConversions: a tenant has conversion
  failures and no conversions for 1h.
- CortexParquetConverterNotConverting: a tenant has blocks that are due
  for conversion and no conversions for 3h.
- CortexParquetConversionDelayHigh: more than 20% of the conversions
  finish more than 5h after the block upload.
- CortexParquetConverterRingUnhealthy: the parquet-converter ring has
  unhealthy members.
- CortexParquetQueryableMostlyTSDB: more than 70% of the blocks that
  the parquet queryable finds are TSDB blocks.
- CortexParquetCompactionStuckAtFirstRange: more than 80% of the
  blocks of a tenant are neither parquet blocks nor due for conversion.
  The converter only takes blocks wider than the first block range, so
  when the compactor cannot compact a tenant the other alerts stay
  quiet while almost nothing is converted.

Add parquet_converter to job_names and instance_names in the mixin
config.

Signed-off-by: Friedrich Gonzalez <1517449+friedrichg@users.noreply.github.com>
@friedrichg
friedrichg force-pushed the parquet-converter-resources-dashboard branch from f1fa10b to ca3f868 Compare October 16, 2026 03:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant