Data Whimsy
  • Home
  • What is Matomo
  • Contact
  • Custom Reports
  • Connect Matomo to other software
  • CLI API etc
  • Matomo Tag Manager Simple Guide
  • Lyn Only
  • Troubleshooting + Deploying Matomo
  • Weird Data
  • Why is data in Matomo giving different numbers than Google Analytics
  • The Dashboard
  • Moving the Data
  • Book Club Forum

Without the dashboard

The top slot of your browser can send requests to Tracking API & Reporting API
The command line of your browser can accept direct Matomo input (e.g. the PHP ./console command).
On this page are a variety of things you can do without ever touching the visual dashboard (Matomo GUI)

Examples in Matomo On Premises of core:invalidate-report-data & core:archive

3/20/2024

0 Comments

 
PART ONE
This part is optional. 
If you don

't need to invalidate reports, skip to Part Two

sudo su www-data -s /bin/bash -c  "php /var/www/littlefurnace.com/matomo/console core:invalidate-report-data"

Matomo parameters are not well-standardized. Here are examples of the syntax for all of the invalidate-report-data parameters I have used this year:
--dates=2024-04-20,today   <---- contrast this to the force-date-range used above
--dates=2024-08-04,2024-08-05
--dates=yesterday 
--periods=day,week
--segment=33
--sites=1,5

Putting it all together, you could run invalidation like this if you wanted to
sudo su www-data -s /bin/bash -c  "php /var/www/littlefurnace.com/matomo/console core:invalidate-report-data --dates=2024-08-04,2024-08-05 --periods=year --segment=33 --sites=1,5"




PART TWO
These are self explanatory hopefully.

If you are in the directory on your machine where the matomo files are, you might be able to just run this, if you are signed in as the right user:
console core:archive

The owner is usually www-data or apache so I usually run it like this:
sudo su www-data -s /bin/bash -c  "php -d memory_limit=3G /var/www/littlefurnace.com/matomo/console core:archive"

Here are examples of the syntax all of the parameters I have used this year:
--concurrent-archivers=1 <---- often won't work; minimum of 2, on many systems
--concurrent-archivers=5
--concurrent-requests-per-website=1
--concurrent-requests-per-website=3
--disable-scheduled-tasks
--force-date-last-n=220
--force-date-range="2023-01-01",today
--force-date-range=2023-09-01
--force-date-range=2023-09-01,2023-09-01
--force-date-range=yesterday,today
--force-idsegments=12
--force-idsites=2,3,11
--force-sites=2
--skip-all-segments
--vvv

I have never run all at the same time, but you could put most of that together and run the following. (I left out the two segment options which would contradict each other if run at the same time, although in that case, the last parameter entered would possibly be the boss ... later parameters take priority over earlier ones usually.)

sudo su www-data -s /bin/bash -c  "php -d memory_limit=6G /var/www/littlefurnace.com/matomo/console core:archive --concurrent-archivers=2 --concurrent-requests-per-website=3 --disable-scheduled-tasks --force-date-last-n=220 --force-idsites=2,3,11 --skip-all-segments --vvv"

​
PART THREE
My personal Matomo is running on a underresourced machine: one CPU, 4G of RAM. So I get to see all of the error messages, as follows.
sudo su www-data -s /bin/bash -c  "php ./console core:archive --force-idsites=1  --force-date-last-n=20"
If you run core:invalidate-report-data and then run the above command, twice in rapid succession, nothing much is done the second time. Each step took 3 seconds. 

sudo su www-data -s /bin/bash -c  "php ./console core:archive --force-idsites=2  --force-date-last-n=1"
Crashed ("unserialization error" and " Warning - shell_exec(): Unable to execute 'ps x 2>/dev/null" and "Unable to execute 'ps aux'"). Half of these mentioned a segment.
sudo su www-data -s /bin/bash -c  "php -d memory_limit=9G  /var/www/littlefurnace.com/matomo/console  core:archive   --concurrent-requests-per-website=1 --concurrent-archivers=3 --skip-all-segments --force-idsites=2"
Succeeded.
Ran all of the Custom Reports, back til the beginning of time for the new CR I authored recently.
sudo su www-data -s /bin/bash -c  "php -d memory_limit=9G  /var/www/littlefurnace.com/matomo/console  core:archive   --concurrent-requests-per-website=1 --concurrent-archivers=3  --force-idsites=2"
Succeeded.
I think the segments worked because I had already run all the basic reports.

PART FOUR
Digging into the SQL database, to audit what is going on with your archiver invalidations.

Your Matomo interface should show hollow points when data is either incomplete (like for today, because today is still in flux IRL) or because the period was invalidated but has not succesfully executed in the core:archive process. 

Will hollow data points persist in the evolution graph after everything is archived? No. Once archiving has recreated the reports both the tooltip ("invalidated") and the hollowness should go away.

How does Matomo know to make a hollow data point in a graph? If
name = "done" && value = 4 is stored in the archive_numeric table, it will still be shown as invalidated in a graph. The archive_invalidations table has no role in that visualization.

PART FIVE 

You can keep track, in your SQL database what has been invalidated.  For example here is some output from my terminal, when there are no invalid reports, and then after I have manually labeled some reports as invalid.

mysql> SELECT * FROM matomo_archive_invalidations;
Empty set (0.00 sec)
$ sudo su www-data -s /bin/bash -c "php /var/www/littlefurnace.com/matomo/console core:invalidate-report-data --sites=1 --dates=2025-04-15"
[sudo] password for evan:
INFO [2025-05-14 19:15:05] 3886 Invalidating day periods in 2025-04-15 for site = [ 1 ], segment = [ ]...
INFO [2025-05-14 19:15:05] 3886 Invalidating day periods in 2025-04-15 for site = [ 1 ], segment = [ visitIp!=91.90.123.187 ]...
INFO [2025-05-14 19:15:05] 3886 Invalidating day periods in 2025-04-15 for site = [ 1 ], segment = [ countryName==China,countryName==taiwan,countryName==Russia ]...
INFO [2025-05-14 19:15:05] 3886 Invalidating week periods in 2025-04-15 for site = [ 1 ], segment = [ ]...
INFO [2025-05-14 19:15:05] 3886 Invalidating week periods in 2025-04-15 for site = [ 1 ], segment = [ visitIp!=91.90.123.187 ]...
INFO [2025-05-14 19:15:05] 3886 Invalidating week periods in 2025-04-15 for site = [ 1 ], segment = [ countryName==China,countryName==taiwan,countryName==Russia ]...
INFO [2025-05-14 19:15:05] 3886 Invalidating month periods in 2025-04-15 for site = [ 1 ], segment = [ ]...
INFO [2025-05-14 19:15:05] 3886 Invalidating month periods in 2025-04-15 for site = [ 1 ], segment = [ visitIp!=91.90.123.187 ]...
INFO [2025-05-14 19:15:05] 3886 Invalidating month periods in 2025-04-15 for site = [ 1 ], segment = [ countryName==China,countryName==taiwan,countryName==Russia ]...
INFO [2025-05-14 19:15:05] 3886 Invalidating year periods in 2025-04-15 for site = [ 1 ], segment = [ ]...
INFO [2025-05-14 19:15:05] 3886 Invalidating year periods in 2025-04-15 for site = [ 1 ], segment = [ visitIp!=91.90.123.187 ]...
INFO [2025-05-14 19:15:05] 3886 Invalidating year periods in 2025-04-15 for site = [ 1 ], segment = [ countryName==China,countryName==taiwan,countryName==Russia ]...


mysql> SELECT * FROM matomo_archive_invalidations ; +----------------+-----------+--------------------------------------+--------+------------+------------+--------+---------------------+------------+--------+--------+-----------------+------------+
| idinvalidation | idarchive | name | idsite | date1 | date2 | period | ts_invalidated | ts_started | status | report | processing_host | process_id |
+----------------+-----------+--------------------------------------+--------+------------+------------+--------+---------------------+------------+--------+--------+-----------------+------------+
| 38300 | NULL | done | 1 | 2025-04-15 | 2025-04-15 | 1 | 2025-05-14 19:15:05 | NULL | 0 | NULL | NULL | NULL |
| 38301 | NULL | done | 1 | 2025-04-14 | 2025-04-20 | 2 | 2025-05-14 19:15:05 | NULL | 0 | NULL | NULL | NULL |
| 38302 | 414 | done | 1 | 2025-04-01 | 2025-04-30 | 3 | 2025-05-14 19:15:05 | NULL | 0 | NULL | NULL | NULL |
| 38303 | 8258 | done | 1 | 2025-01-01 | 2025-12-31 | 4 | 2025-05-14 19:15:05 | NULL | 0 | NULL | NULL | NULL |
| 38304 | NULL | donec1f9075b020fe8e4a5e7ebae74c38e76 | 1 | 2025-04-15 | 2025-04-15 | 1 | 2025-05-14 19:15:05 | NULL | 0 | NULL | NULL | NULL |
| 38305 | NULL | donec1f9075b020fe8e4a5e7ebae74c38e76 | 1 | 2025-04-14 | 2025-04-20 | 2 | 2025-05-14 19:15:05 | NULL | 0 | NULL | NULL | NULL |
| 38306 | 424 | donec1f9075b020fe8e4a5e7ebae74c38e76 | 1 | 2025-04-01 | 2025-04-30 | 3 | 2025-05-14 19:15:05 | NULL | 0 | NULL | NULL | NULL |
| 38307 | 8266 | donec1f9075b020fe8e4a5e7ebae74c38e76 | 1 | 2025-01-01 | 2025-12-31 | 4 | 2025-05-14 19:15:05 | NULL | 0 | NULL | NULL | NULL |
| 38308 | NULL | done5d6556a3a658d05f4bf2bd19fa1e7daf | 1 | 2025-04-15 | 2025-04-15 | 1 | 2025-05-14 19:15:05 | NULL | 0 | NULL | NULL | NULL |
| 38309 | NULL | done5d6556a3a658d05f4bf2bd19fa1e7daf | 1 | 2025-04-14 | 2025-04-20 | 2 | 2025-05-14 19:15:05 | NULL | 0 | NULL | NULL | NULL |
| 38310 | 425 | done5d6556a3a658d05f4bf2bd19fa1e7daf | 1 | 2025-04-01 | 2025-04-30 | 3 | 2025-05-14 19:15:05 | NULL | 0 | NULL | NULL | NULL |
| 38311 | 8265 | done5d6556a3a658d05f4bf2bd19fa1e7daf | 1 | 2025-01-01 | 2025-12-31 | 4 | 2025-05-14 19:15:05 | NULL | 0 | NULL | NULL | NULL |
+----------------+-----------+--------------------------------------+--------+------------+------------+--------+---------------------+------------+--------+--------+-----------------+------------+
12 rows in set (0.00 sec)







0 Comments



Leave a Reply.

    Author

    Evan Genest

    Archives

    November 2024
    August 2024
    May 2024
    March 2024
    February 2024
    November 2023
    October 2023
    September 2023
    August 2023

    Categories

    All

Powered by Create your own unique website with customizable templates.
  • Home
  • What is Matomo
  • Contact
  • Custom Reports
  • Connect Matomo to other software
  • CLI API etc
  • Matomo Tag Manager Simple Guide
  • Lyn Only
  • Troubleshooting + Deploying Matomo
  • Weird Data
  • Why is data in Matomo giving different numbers than Google Analytics
  • The Dashboard
  • Moving the Data
  • Book Club Forum