Repository navigation
Add parquet dashboards and alerts - #68
Open
friedrichg wants to merge 2 commits into
Open
friedrichg wants to merge 2 commits into
friedrichg wants to merge 2 commits into
Conversation
friedrichg
force-pushed
the
parquet-converter-resources-dashboard
branch
from
October 16, 2026 03:48
b4ffd3f to
f1fa10b
Compare
The parquet converter downloads whole TSDB blocks to local disk and converts them to parquet, and the queriers and rulers read the parquet files through the parquet queryable. Add dashboards and alerts for this pipeline. Cortex / Parquet Converter Resources has the same rows and panels as Cortex / Compactor Resources (CPU and memory, network, disk), for the parquet-converter container. Cortex / Parquet shows the pipeline end to end, in the same layout as Cortex / Compactor: - Converter: block conversions and failures, tenants owned, the conversion delay histogram and the latest conversion duration. - Bucket: the tenants with the most unconverted blocks and the most parquet blocks, from the compactor's bucket index gauges. - Querier and Ruler: blocks queried and operations by type (parquet, tsdb, mixed), and the parquet shard cache hit ratio and items. - Object Store: the parquet-converter bucket client. The cortex_parquet_alerts group, with a playbook entry for each alert: - CortexParquetConverterFailingConversions: a tenant has conversion failures and no conversions for 1h. - CortexParquetConverterNotConverting: a tenant has blocks that are due for conversion and no conversions for 3h. - CortexParquetConversionDelayHigh: more than 20% of the conversions finish more than 5h after the block upload. - CortexParquetConverterRingUnhealthy: the parquet-converter ring has unhealthy members. - CortexParquetQueryableMostlyTSDB: more than 70% of the blocks that the parquet queryable finds are TSDB blocks. - CortexParquetCompactionStuckAtFirstRange: more than 80% of the blocks of a tenant are neither parquet blocks nor due for conversion. The converter only takes blocks wider than the first block range, so when the compactor cannot compact a tenant the other alerts stay quiet while almost nothing is converted. Add parquet_converter to job_names and instance_names in the mixin config. Signed-off-by: Friedrich Gonzalez <1517449+friedrichg@users.noreply.github.com>
friedrichg
force-pushed
the
parquet-converter-resources-dashboard
branch
from
October 16, 2026 03:52
f1fa10b to
ca3f868
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does:
Adds two dashboards and an alert group for the parquet converter and the parquet queryable.
Cortex / Parquet, in the same layout asCortex / Compactor:cortex_parquet_converter_blocks_converted_total,cortex_parquet_converter_block_convert_failures_total), tenants owned per instance, the conversion delay histogram (cortex_parquet_converter_convert_block_delay_minutes), and the latest conversion duration per tenant (a gauge, top 10).cortex_bucket_parquet_unconverted_blocks_count,cortex_bucket_parquet_blocks_count, reported by the compactor).type(parquet,tsdb,mixed), and the parquet shard cache hit ratio and item count.parquet-converterbucket client.All metrics exist in Cortex v1.21.1.
Cortex / Parquet Converter Resources, with the same rows and panels asCortex / Compactor Resources, for theparquet-convertercontainer:The parquet converter downloads whole TSDB blocks to its local disk before it converts them, so its disk and network use need the same view as the compactor.
Alerts (
cortex_parquet_alerts, with a playbook entry for each):CortexParquetConverterFailingConversionsCortexParquetConverterNotConvertingCortexParquetConversionDelayHighCortexParquetConverterRingUnhealthyparquet-converterring has unhealthy members for 15mCortexParquetQueryableMostlyTSDBCortexParquetCompactionStuckAtFirstRangeThe last alert covers a case that the parquet metrics alone cannot show. The converter only takes blocks wider than the first
-compactor.block-rangesrange, so when the compactor cannot compact a tenant,cortex_bucket_parquet_unconverted_blocks_countstays 0 and almost nothing is converted.Also adds
parquet_converter: 'parquet-converter.*'tojob_namesandinstance_namesin the mixin config.Checks:
make lint-mixin-with-jsonnetfmtpasses.make build-mixingeneratescortex-parquet.jsonandcortex-parquet-converter-resources.json.make lint-playbookspasses. The alerts add nomixtool lintfindings.mixtool lintreports 936 findings against 838 onmain. The new findings are the sametarget-instance-rule,target-job-rule,template-datasource-ruleandtemplate-job-rulefindings thatCortex / Compactor(78) andCortex / Compactor Resources(22) already have, because the dashboards use the same helpers.limitlines (the converter there has no CPU or memory limits) and the failure counter (no failed conversions). The six alert expressions also run there. OnlyCortexParquetCompactionStuckAtFirstRangefires, for a 100M-series tenant whose compactors cannot finish the first-range merge (about 10,000 2h blocks and 2 parquet blocks).Checklist
CHANGELOG.mdupdated