T2912 - Counters Next Gen
===
## Current
![current architecture](https://forge.softwareheritage.org/file/data/rf5575cnlldxxutbse2w/PHID-FILE-hsgp6vtc42p6jjm274qs/Object_counters_pipeline.png)
Source: [counters.plantuml](https://archive.softwareheritage.org/swh:1:cnt:daa576518962c4dab23a88f44b5d5f5416bd2a11;origin=https://forge.softwareheritage.org/source/snippets.git;visit=swh:1:snp:42b064f9ffaad7183118a17b3fea2b4cba936e8e;anchor=swh:1:rev:194a686fc4043ea20be0609363d1b0ca4d2567c8;path=/vsellier/plantuml/T2912-counters/counters.plantuml/)
## Target proposal
![Next generation](https://hedgedoc.softwareheritage.org/uploads/upload_bbd117db42ad411183bb97ea351f9911.png)
Source: [counters-nextgen.plantuml](https://archive.softwareheritage.org/swh:1:cnt:e856faf084fa5bb0033068dd87fa32952a7b35c1;origin=https://forge.softwareheritage.org/source/snippets.git;visit=swh:1:snp:42b064f9ffaad7183118a17b3fea2b4cba936e8e;anchor=swh:1:rev:194a686fc4043ea20be0609363d1b0ca4d2567c8;path=/vsellier/plantuml/T2912-counters/counters-nextgen.plantuml/)
## Plan
- Create a new swh-counters module (name to be defined/improved)
- Create a journal client to populate the counters
- Create a gunicorn service to expose the counters and prometheus metrics
- when the counter is up-to-date, activate the prometheus exporter
- [Reply question 2]
- update the `export_archive_counters.py` to use the new metrics
- Plug the '/api/1/stat/counters/' of the webapp to the counter service instead of storage
- [later when battle tested] Cleanup swh-storage (scripts/code/psql/database tables)
## Questions
1. Which metric name to use? proposal: `swh_archive_object_count(type="")` (to match the current naming: `sql_swh_archive_object_count`)
2. how to keep the current history? (*preference to 2*)
1. export the current stats to a static json as for munin and merge the 2 in `export_archive_counters.py`
2. export the current stats to a static json, merge it with the munin data into a new file and use it in `export_archive_counters.py`
3. compute the history from the munin data + sql_swh_archive_object_count time series + new timeserie in `export_archive_counters.py`?