In a previous post I explained how to instal and configure convirt2 in a Debian GNU/Linux environment.
As your virtualisation infrastruture grows, the default parameter ofconvirt may become inappropriate, and you may find some issue like increase of latency (more specifically in SQL statistics collection), gaps in graphs or error messages (mostly reported by mail, if SMTP is correctly configured) like the following ones:
Task failed at 2011-07-13 06:44:40 Task submission failed because all worker threads are occupied. Please increase the number of threads.
or
Update node availability Task failed at 2011-07-28 18:06:12 (OperationalError) (1205, 'Lock wait timeout exceeded; try restarting transaction') u'INSERT INTO cms_locks (sub_system, entity_id, operation, table_name) VALUES (%s, %s, %s, %s)' ['AvailState', '9a893ea6-ebe1-1ff4-8a5a-dfae235979de', 'MANAGED_NODE', 'avail_current']
or
The transaction is inactive due to a rollback in a subtransaction. Issue rollback() to cancel the transaction.
or
Collect Metrics Task failed at 2011-07-28 18:01:37 (OperationalError) (1205, 'Lock wait timeout exceeded; try restarting transaction') u'INSERT INTO cms_locks (sub_system, entity_id, operation, table_name) VALUES (%s, %s, %s, %s)' ['Metrics', '9a893ea6-ebe1-1ff4-8a5a-dfae235979de', 'CollectMetrics', 'metrics/metrics_curr']
Here are some hints that will hopefully make your convirt install run smoothly again.
First of all, stop convirt and backup your database!
Fix the locking system:
Grab the patch provided by convirt http://www.convirture.com/forums/download/file.php?id=300 and apply it as follow:
# be convirt user or whatever user you use to run it
convirt@cms:~$ cd /var/lib/convirt # or cd to whatever convirt working directory you're using
convirt@cms:~$ wget http://www.convirture.com/forums/download/file.php?id=300
convirt@cms:~$ mv file.php?id=300 cms_lock.diff
convirt@cms:~$ patch -p0 < cms_lock.diff
Purge data
If you are installing convirt from scratch and have no data collected yet, you can skip this step and thank your favorite spiritual entity for not loosing too much time. If you had convirt running for a while with default parameter, you certainly keep a huge amount of unused data in your database that slow down the whole process. So here, we’re going to get rid of all that stuffs.
Unfortunately selectively deleting the unused data require indexing first. Index creation is a looooong process if you have many data collected yet, but deleting old data without such an index is even longer… so be patient.
convirt@cms:~$ mysql -u root -p convirt
mysql> alter table metrics add index `m_rype_cdate` (`metric_type`,`cdate`)
Let’s purge data now. First we need to define a time period before which we consider we don’t need to have precise metrics about our hosts and VMs. Do you care of the precise CPU usage of each machine at 00:32:34 4 month ago? Certainly not. More generally I consider there is no use to keep non consolidated data that are older than 2 weeks.
Then grab another script from the convirt forum and do what follow:
convirt@cms:~$ wget http://www.convirture.com/forums/download/file.php?id=317
convirt@cms:~$ mv file.php?id=317 cleanup_metrics.sh.gz
convirt@cms:~$ gzip -d cleanup_metrics.sh.gz
convirt@cms:~$ ./cleanup_metrics.sh 15 days -u root -p convirt # refer to the following post for details on syntax
We can now destroy the big index that helped us deleting data:
ALTER table `metrics` DROP INDEX `m_rype_cdate`;
Modifying default config
Modify src/convirt/web/convirt/development.ini as follow:
services.execution.num_threads = 50
purge_raw_data = 15
SQL Indexes
The creation of the following indexes speeds up data collection and avoid rollback and thread problems:
ALTER table `cms_locks` ADD INDEX `cmsl_subsys_eid_ope_tname` (`sub_system`, `entity_id`, `operation`, `table_name`);
ALTER table `metrics` ADD INDEX `eid_cdate` (`entity_id`, `cdate`);
Restart convirt, which should now run much more smoothly.
Leave a Reply