|
So, you're facing an issue with the Matomo Queued Tracking plugin, where large request sets are failing to process. Maybe your error indicates a rollback occurred because the process could not acquire a lock. This situation can arise due to configuration issues or contention for resources.
Here’s how you can troubleshoot and resolve the issue: --- ### 1. **Check Lock Status** The command provided in the error message helps verify whether a lock is active: ```bash php /var/www/html/matomo/console queuedtracking:lock-status ``` Run this command to check if any other processes are holding a lock. If a lock is stuck, you may need to clear it manually. --- ### 2. **Manually Clear the Lock** If a stuck lock is detected, you can clear it by running: ```bash php /var/www/html/matomo/console queuedtracking:unlock ``` This command ensures no stale locks block the process. --- ### 3. **Optimize the Request Set Size** Having over 200,000 requests in a single set can cause memory and processing issues. Reduce the batch size by configuring the `maxRequestsPerQueue` setting in `config/config.ini.php`: ```ini [QueuedTracking] maxRequestsPerQueue = 50000 ``` Restart Matomo after making this change to ensure it takes effect. --- ### 4. **Increase Resources** Your log indicates memory usage peaking at 15.64 GB. If this is close to your system’s limit, the process might fail due to insufficient resources. Consider increasing: - PHP memory limit in `php.ini`: ```ini memory_limit = 32G ``` - Worker processes (use with caution, especially if resources are constrained): ```bash php /var/www/html/matomo/console queuedtracking:process --workers=4 ``` --- ### 5. **Process Smaller Batches** Process fewer request sets at a time by adding a limit to the processing command: ```bash php /var/www/html/matomo/console queuedtracking:process --queue-id=0 --force-num-requests-process-at-once=10000 ``` This ensures that requests are handled in smaller chunks, reducing the load on your system. --- ### 6. **Monitor Logs** Review Matomo logs for detailed error messages: ```bash tail -f /var/www/html/matomo/tmp/logs/matomo.log ``` Identify if there are issues related to specific request sets or database performance. --- ### 7. **Database Tuning** Since queued tracking relies heavily on the database, ensure your database is optimized: - Increase `innodb_buffer_pool_size` in your MySQL configuration. - Ensure indexes on the `log_*` tables are optimized. - Regularly archive old data using Matomo’s cron jobs. --- ### 8. **Restart Services** After making changes, restart necessary services: ```bash sudo service php8.3-fpm restart sudo service mysql restart ``` --- ### Final Step After applying these changes, re-run the processing command: ```bash php /var/www/html/matomo/console queuedtracking:process ``` --- These steps should help resolve your issue. Let me know if you need further assistance!
0 Comments
You can adjust MySQL's memory usage by modifying the my.cnf configuration file. For example, to double the memory allocated, you'll likely need to adjust key parameters in the configuration file that control memory usage. Here are the steps:
Key Parameters to Modify in my.cnf
Restart MySQL for Changes to Take EffectRestart MySQL to apply the changes: sudo systemctl restart mysql Monitor Memory UsageAfter restarting, monitor the memory usage to ensure the settings have taken effect: mysql -e "SHOW VARIABLES LIKE 'innodb_buffer_pool_size';" mysql -e "SHOW STATUS LIKE 'Innodb_buffer_pool_bytes_data';" Additionally, use top or htop to verify the overall memory usage. Important Notes
 In cookieless mode, Matomo works great. Cookieless mode cannot detect the return of the visitor well in that mode but Matomo still gives you 90% of the data you need otherwise.
The main cookie: pk_id Out of the box (the first time you use it), Matomo's default configuration (1) will set a pk_id cookie. By default the cookie expires in 13 months. You can see the pk_id cookie right now: try the following. Open the homepage of Matomo.org (screenshot above). In the browser devtools find pk_id. It is a stringified array. The array holds two things: visitorID of the person and the date the cookie was made. config_id The backup plan, if there is no pk_id cookie found in the browser, nor user_id found, will be the "not a fingerprint" number. Link 1 Matomo Default Configuration: github.com/matomo-org/matomo/blob/5.x-dev/config/global.ini.php I hate getting these messages when running core:archive in Matomo.
My Matomo server is a $5 shared space hosted on Digital Ocean, the cheapest deployment, the least memory, all minimal RAM, CPUs, everything. Frustrated by these errors, I learned to overcome these errors by running console core:archive manually and changing options until it runs well. Here are the things I do. 1. Incrementally run longer time spans. I run today and yesterday. Then this month. Then forever. 2. Run it once with no segments. Then run it with segments. 3. Run it with the PHP memory allocation declared at run time. Case 1: Got errors during the archiving, in the section before scheduled tasks. Added the option --skip-all-segments. Still errors. Deleted the segment from the UI. Success! (Not sure why the option didn't do the trick.) Case 2 (September 2024 update) 1. Tons of errors: sudo su www-data -s /bin/bash -c "php -d memory_limit=8G /var/www/matomo/console core:archive" 2. Zero errors: sudo su www-data -s /bin/bash -c "php -d memory_limit=8G /var/www/littlefurnace.com/matomo/console core:archive --skip-all-segments --force-idsites=2" Conclusion: On underpowered servers, you can eliminate errors by doing one site at a time and by skipping segments." Then go back and do the segments, still doing only one site at a time 3. sudo su www-data -s /bin/bash -c "php -d memory_limit=8G /var/www/littlefurnace.com/matomo/console core:archive --force-idsites=1" Do you want to see cookies appearing and disappearing, and expiry times changing, in your browser?
For setting Cookie Consent (this is the yes/no after a visitor clicks your "accept all cookies" banner):
There is an entire parallel process for tracking in general (not just cookies). Repeat every line above the same but remove the word cookie from methods in push. Links Easy: https://matomo.org/faq/new-to-piwik/how-can-i-still-track-a-visitor-without-cookies-even-if-they-decline-the-cookie-consent/ More detail: https://developer.matomo.org/guides/tracking-consent List of most of the cookies: https://matomo.org/faq/general/faq_146/ List of some innocent cookies: https://matomo.org/faq/general/faq_157/ ALL 200 JS METHODS FOR THE CONSOLE: https://developer.matomo.org/api-reference/tracking-javascript Errors which say Something happened, Unable to create, or Unable to write are caused by lack of permission. Your server user-name is usually www-data or sometimes apache.
To fix this error do all of the steps, in order.
 Tokens generated by users, starting in Matomo 5, should keep unchecked the box "Only allow secure requests" if the token is meant for a GET request. However it is recommended that you check this box, and get in the habit of sending tokens in POST requests instead.
Certain operations by modules like Looker Studio Matomo Connector require a GET request from Google. So in that case, generate the token with the box NOT checked. If your site gets more than 100,000 pageviews per month, you must, must, must enable core:archive. And you must disable browser based archiving. This can be done in the config/config.ini.php file.
How long does core:archive take to run A very large site with 12M hits per day (300M hits per month) can take 8 hours to finish the core:archive. This is with some CPU-intensive options enabled, like uniqueness and segments. Here is one example of a real site which takes 8 hours:
Is PHP installed?
sudo apt update sudo apt install php There are two! Matomo runs two PHPs. One is on the Apache server for the Matomo backend. The other is the PHP for the CLI, which executes the core:archive command PART ONE: JUST CHECKING THE SERVER PHP WHERE REPORTS AND TRACKING HAPPEN Check memory and the version of PHP in the web server which serves your Matomo Interface First, make a serveable phpinfo(); page. I placed mine at https://littlefurnace.com/foobarbaz.php Then serve that page, in the browser. Or from CURL. In Curl: curl https://littlefurnace.com/foobarbaz.php | grep memory In your browser, the classic pink and blue PHP information page will appear. This will have important information about the memory_limit, the path to where the configuration files are, and the PHP version you are running. PART TWO: JUST CHECKING THE CLI ENVIRONMENT PHP, WHERE CORE:ARCHIVE AND OTHER CONSOLE COMMANDS ARE EXECUTED Check memory and the version of PHP in CLI (for Matomo core:archive) IMPORTANT The core:archive of Matomo uses the PHP settings of CLI, not of the web server. php -i | grep memory_limit To find the path to the config file where you can edit things like memory_limit php -i | grep -i config Example in my Matomo CLI environment This: php -i | grep -i config Returns this: Loaded Configuration File => /etc/php/8.1/cli/php.ini So edit that file to look like this: memory_limit=6G Is PHP enabled for Apache? sudo apt install libapache2-mod-php (restart Apache) sudo systemctl restart apache2 Configuration Check the PHP configuration file (php.ini) to ensure that the disable_functions directive does not include any module that you need! Find the php.ini php -i | grep php.ini Then, edit the php.ini file And restart server sudo systemctl restart apache2 Increase PHP memory in Matomo A typical Matomo error is Allowed memory size of 4294967296 bytes exhausted Typical steps to fix this error: Find the php.ini php -i | grep php.ini Then, edit this line in the php.ini file memory_limit=8G And restart server sudo systemctl restart apache2 (Or, if you want to experiment on-the-fly with benchmarking the performance, you can actually override the memory limit directly, including in the crontab, using an option such as: php -d memory_limit=2G /var/www/html/console core:archive ) A less common, laborious, alternative solution to prevent the php process from using too much memory is only process a specific number of reports at a time in order. We can achieve this by using a combination of --force-periods and --force-date-range in the core:archive task to process only a few specific reports for each core:archive run: For example: php path/to/matomo/console core:archive --force-idsite=13 --force-periods=day,week,month --force-date-range=2021-01-01,2021-01-31 And then repeat this process for each month that the reports were invalidated for. If you don't think data is arriving in your Matomo dashboard, here are three sequential types of troubleshooting: 1) Type 1: Test your website in the browser. Load some specific pages you are monitoring into Firefox, Chrome, or similar. Watch browser >> developer tools >> network tab. Confirm that a tracking signal goes to Matomo. 2) Type 2: As you navigate your website, watch in the Matomo dashboard: Matomo >> Visitors >> Visits log. This is the raw, unenriched, unmanipulated data. Confirm that all of your website traffic arrives there. 3) Type 3: If everything looks good in 1 and , 2, now for your troubleshooting, check the numbers in the rest of Matomo (this is the enriched, manipulated data, after matomo's backend php modifies it). Look especially at matomo >> Behavior >> Pages maybe.  Type 1: Test your website in the browser. The Matomo pageview event can be manually observed in your browser devtools. In the case above we have configured a Javascript tracker on our website (pikl.us website) to monitor pageviews. The deluxe twist is we are sending the pageview to two Matomo servers. We set this up using the instructions here: https://matomo.org/faq/how-to/how-do-i-send-tracking-requests-to-two-or-more-matomo-servers/ Since it is easy (and catastrophic to data) to miss a pageview event, we want to confirm that we set it up correctly. (It's a common mistake to set up two servers and later discover that the data is uneven because you are only sending to the most recently set server.) Let's troubleshoot this in Developer Tools in the browser. To confirm the pageviews are actually happening, do the following:
In my case, the POST request contains the following:  https://tatll.matomo.cloud/matomo.php?action_name=pikL&
idsite=6& rec=1& r=333268& h=9& m=13& s=44& url=https://pikl.us/linksPage.php& _id=675de8e438add64b& _idn=0& send_image=0& _refts=0& pdf=1& qt=0& realp=0& wma=0& fla=0& java=0& ag=0& cookie=1& res=2560x1080& pv_id=U80TLm& pf_net=292& pf_srv=72& pf_tfr=0& uadata={} If you would like to learn more about the parameters in the request, see the Matomo parameters documentation here: https://developer.matomo.org/api-reference/tracking-api TODO: Specific examples of troubleshooting, Type 2 and Type 3, will be added here in the future. |
