Data Whimsy
  • Home
  • What is Matomo
  • Contact
  • Custom Reports
  • Connect Matomo to other software
  • CLI API etc
  • Matomo Tag Manager Simple Guide
  • Lyn Only
  • Troubleshooting + Deploying Matomo
  • Weird Data
  • Why is data in Matomo giving different numbers than Google Analytics
  • The Dashboard
  • Moving the Data
  • Book Club Forum

Queued Tracking Troubleshooting in Matomo

12/16/2024

0 Comments

 
So, you're facing an issue with the Matomo Queued Tracking plugin, where large request sets are failing to process. Maybe your error indicates a rollback occurred because the process could not acquire a lock. This situation can arise due to configuration issues or contention for resources.
Here’s how you can troubleshoot and resolve the issue:
---
### 1. **Check Lock Status**
The command provided in the error message helps verify whether a lock is active:
```bash
php /var/www/html/matomo/console queuedtracking:lock-status
```
Run this command to check if any other processes are holding a lock. If a lock is stuck, you may need to clear it manually.
---
### 2. **Manually Clear the Lock**
If a stuck lock is detected, you can clear it by running:
```bash
php /var/www/html/matomo/console queuedtracking:unlock
```
This command ensures no stale locks block the process.
---
### 3. **Optimize the Request Set Size**
Having over 200,000 requests in a single set can cause memory and processing issues. Reduce the batch size by configuring the `maxRequestsPerQueue` setting in `config/config.ini.php`:
```ini
[QueuedTracking]
maxRequestsPerQueue = 50000
```
Restart Matomo after making this change to ensure it takes effect.
---
### 4. **Increase Resources**
Your log indicates memory usage peaking at 15.64 GB. If this is close to your system’s limit, the process might fail due to insufficient resources. Consider increasing:
- PHP memory limit in `php.ini`:
```ini
memory_limit = 32G
```
- Worker processes (use with caution, especially if resources are constrained):
```bash
php /var/www/html/matomo/console queuedtracking:process --workers=4
```
---
### 5. **Process Smaller Batches**
Process fewer request sets at a time by adding a limit to the processing command:
```bash
php /var/www/html/matomo/console queuedtracking:process --queue-id=0 --force-num-requests-process-at-once=10000
```
This ensures that requests are handled in smaller chunks, reducing the load on your system.
---
### 6. **Monitor Logs**
Review Matomo logs for detailed error messages:
```bash
tail -f /var/www/html/matomo/tmp/logs/matomo.log
```
Identify if there are issues related to specific request sets or database performance.
---
### 7. **Database Tuning**
Since queued tracking relies heavily on the database, ensure your database is optimized:
- Increase `innodb_buffer_pool_size` in your MySQL configuration.
- Ensure indexes on the `log_*` tables are optimized.
- Regularly archive old data using Matomo’s cron jobs.
---
### 8. **Restart Services**
After making changes, restart necessary services:
```bash
sudo service php8.3-fpm restart
sudo service mysql restart
```
---
### Final Step
After applying these changes, re-run the processing command:
```bash
php /var/www/html/matomo/console queuedtracking:process
```
---
These steps should help resolve your issue. Let me know if you need further assistance!
0 Comments

SQL Database Memory Settings in my.cnf

12/10/2024

0 Comments

 
You can adjust MySQL's memory usage by modifying the my.cnf configuration file. For example, to double the memory allocated, you'll likely need to adjust key parameters in the configuration file that control memory usage. Here are the steps:


Key Parameters to Modify in my.cnf
  1. Locate my.cnf: Typically found in one of these locations:
    • /etc/my.cnf
    • /etc/mysql/my.cnf
    • /usr/local/mysql/etc/my.cnf
    If you’re unsure, you can locate it with:
    mysql --help | grep my.cnf
  2. Key Settings to Adjust: These are the most commonly modified settings that influence memory usage:
    • innodb_buffer_pool_size: The largest consumer of memory, used for caching data and indexes for InnoDB tables.
      • Example: To double the memory allocated:
        innodb_buffer_pool_size=1G # Adjust based on your available memory
    • key_buffer_size: Memory used for caching MyISAM table indexes. If you primarily use InnoDB, this can be kept low.
    • query_cache_size: If query caching is enabled, increasing this can also improve performance.
    • max_connections: Each connection consumes memory. If you allow too many concurrent connections, it can inflate memory usage.
    • tmp_table_size and max_heap_table_size: Memory allocated for temporary tables. If you expect many temporary tables, these values may need adjustment.
  3. Example Adjustments:
    [mysqld] innodb_buffer_pool_size=2G key_buffer_size=256M query_cache_size=128M max_connections=200 tmp_table_size=64M max_heap_table_size=64M
  4. Save and Exit: After editing my.cnf, save your changes.

Restart MySQL for Changes to Take EffectRestart MySQL to apply the changes:
sudo systemctl restart mysql
Monitor Memory UsageAfter restarting, monitor the memory usage to ensure the settings have taken effect:
mysql -e "SHOW VARIABLES LIKE 'innodb_buffer_pool_size';" mysql -e "SHOW STATUS LIKE 'Innodb_buffer_pool_bytes_data';" Additionally, use top or htop to verify the overall memory usage.

Important Notes
  • Ensure Available Memory: Verify that doubling MySQL’s memory won’t exhaust your system’s resources. Keep some buffer for the OS and other processes.
  • Test Gradually: Start with smaller increases and monitor performance before doubling the memory outright.
  • Backup Configuration: Save a backup of your my.cnf file before making changes.
Let me know if you’d like more guidance or tailored recommendations based on your server setup!

0 Comments

    Troubleshooting + Deploying Matomo

    Archives

    December 2024
    July 2024
    April 2024
    March 2024
    February 2024
    November 2023
    October 2023

    Categories

    All

Powered by Create your own unique website with customizable templates.
  • Home
  • What is Matomo
  • Contact
  • Custom Reports
  • Connect Matomo to other software
  • CLI API etc
  • Matomo Tag Manager Simple Guide
  • Lyn Only
  • Troubleshooting + Deploying Matomo
  • Weird Data
  • Why is data in Matomo giving different numbers than Google Analytics
  • The Dashboard
  • Moving the Data
  • Book Club Forum