364 views
T2912 - Counters Next Gen === ## Current ![current architecture](https://forge.softwareheritage.org/file/data/rf5575cnlldxxutbse2w/PHID-FILE-hsgp6vtc42p6jjm274qs/Object_counters_pipeline.png) Source: [counters.plantuml](https://archive.softwareheritage.org/swh:1:cnt:daa576518962c4dab23a88f44b5d5f5416bd2a11;origin=https://forge.softwareheritage.org/source/snippets.git;visit=swh:1:snp:42b064f9ffaad7183118a17b3fea2b4cba936e8e;anchor=swh:1:rev:194a686fc4043ea20be0609363d1b0ca4d2567c8;path=/vsellier/plantuml/T2912-counters/counters.plantuml/) ## Target proposal ![Next generation](https://hedgedoc.softwareheritage.org/uploads/upload_bbd117db42ad411183bb97ea351f9911.png) Source: [counters-nextgen.plantuml](https://archive.softwareheritage.org/swh:1:cnt:e856faf084fa5bb0033068dd87fa32952a7b35c1;origin=https://forge.softwareheritage.org/source/snippets.git;visit=swh:1:snp:42b064f9ffaad7183118a17b3fea2b4cba936e8e;anchor=swh:1:rev:194a686fc4043ea20be0609363d1b0ca4d2567c8;path=/vsellier/plantuml/T2912-counters/counters-nextgen.plantuml/) ## Plan - Create a new swh-counters module (name to be defined/improved) - Create a journal client to populate the counters - Create a gunicorn service to expose the counters and prometheus metrics - when the counter is up-to-date, activate the prometheus exporter - [Reply question 2] - update the `export_archive_counters.py` to use the new metrics - Plug the '/api/1/stat/counters/' of the webapp to the counter service instead of storage - [later when battle tested] Cleanup swh-storage (scripts/code/psql/database tables) ## Questions 1. Which metric name to use? proposal: `swh_archive_object_count(type="")` (to match the current naming: `sql_swh_archive_object_count`) 2. how to keep the current history? (*preference to 2*) 1. export the current stats to a static json as for munin and merge the 2 in `export_archive_counters.py` 2. export the current stats to a static json, merge it with the munin data into a new file and use it in `export_archive_counters.py` 3. compute the history from the munin data + sql_swh_archive_object_count time series + new timeserie in `export_archive_counters.py`?