Mcollective, short for “Marionette Collective”, is software written by R.I. Pienaar. The goal is to facilitate the management of numerous machines from a central point. It can be compared to tools like Fabric or Capistrano because it allows you to launch many actions in parallel on remote machines, but it differs on a notable point: it does not rely on SSH. Indeed, the program relies on middleware and provides features that make you not just any admin, but THE ultimate admin.
Why is this? Because a good part of the repetitive and tedious work is handled directly by the program. With the two software mentioned above, you need to know which machines are there and what configuration they carry. In short, you have to keep an up-to-date list. With Mcollective, client discovery is automatic: machines register themselves on a server, and during a request, messages are dispatched to all hosts via the middleware.
Mcollective uses a daemon that runs on each machine. The latter uses agents to perform the various actions expected of it: managing packages, services, or sending messages. Each agent subscribes to a “topic” of the middleware and waits for corresponding messages.
I tried a half Debian 6, half Debian 7 installation to take advantage of ActiveMQ packages, but I don’t recommend it because there are a lot of dependency problems related to the Ruby and gems versions. That’s why I went with Debian 7.
A web interface for MCollective administration has been created: MCOMaster.
Mcollective uses a queue server, so I’ll suggest two options:
Stomp server: designed for small installations
ActiveMQ: a Java powerhouse, but necessary for large installations
Use the one that interests you. Here’s how Mcollective is structured and should work:
The client actually corresponds to the machine that acts as the Mcollective server. It is the node that will control the Mcollective servers.
The Mcollective servers are represented here by nodes in the form of clusters. For a machine to be controlled by the Mcollective client, the server must be installed on it.
The middleware corresponds to the queue server (ActiveMQ for example).
This will create a ‘mcollective’ user with the password ‘marionette’ and give full access (read, write, and admin) to “/topic/mcollective.*”. Adapt the password according to your needs.
We’ll insert a file with the default credentials for ActiveMQ:
Next, we’ll add some options to avoid problems when booting ActiveMQ:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
# Time to wait for the server to start, in secondsSTARTTIME=5# !!! Use a specific data directory for each instanceACTIVEMQ_BASE="/var/lib/activemq/$INSTANCE"# Use openjdk-6 as default Java runtimeJAVA_HOME="/usr/lib/jvm/java-6-openjdk/"# Set some JVM memory optionsACTIVEMQ_OPTS="-Xms512M -Xmx512M -Dorg.apache.activemq.UseDedicatedTaskRunner=true"# Arguments to launch /usr/bin/activemqACTIVEMQ_ARGS="start xbean:activemq.xml"# ActiveMQ configuration filesACTIVEMQ_CONF="/etc/activemq/instances-enabled/$INSTANCE"
Then we will enable this configuration and start ActiveMQ:
You’ll be presented with a first option. If there’s Puppet on this machine, it will ask you to remove it; say no, and a second more flexible proposal will be offered. Accept this one.
> mco helpThe Marionette Collective version 2.0.0
controller Control the mcollective daemon
facts Reports on usage for a specific fact
filemgr Generic File Manager Client
find Find hosts matching criteria
help Application list and help inventory General reporting tool for nodes, collectives and subcollectives
nrpe Client to the Nagios Remote Plugin Execution system
package Install and uninstall software packages
pgrep Distributed Process Management
ping Ping all nodes
plugin MCollective Plugin Application
rpc Generic RPC agent client application
service Start and stop system services
# all machines with the service agentmco ping -A service
mco ping --with-agent service
# all machines with the apache class on themmco ping -C apache
mco ping --with-class apache
# all machines with a class that match the regular expressionmco ping -C /service/
# all machines in the UKmco ping -F country=uk
mco ping --with-fact country=uk
# all machines in either UK or USAmco ping -F "country=/uk|us/"# just the machines called dev1 or dev2mco ping -I dev1 -I dev2
# all machines in the domain foo.commco ping -I /foo.com$/
# all machines with classes matching /apache/ in the UKmco ping -W "/apache/ location=uk"
For even more advanced filters, I recommend the official documentation. Here’s an example of a somewhat complex search:
1
mco service restart httpd -S "((customer=acme and environment=staging) or environment=development) and /apache/"
Modules allow you to add functionality to Mcollective. For example, the management of services, packages, nrpe plugins…
We’ll see here how to install some of them. To make this article simple and compatible with RedHat/Debian, we’ll use variables for plugin installation and create some missing folders:
The Service module allows you to use service management: stop, start, restart, enable, disable and status. We’ll need to install one part on the servers and another part on the client.
For usage it’s simple, just look at my request for the ssh service:
1
2
3
4
5
6
7
8
9
10
11
> mco service ssh status
Do you really want to operate on services unfiltered? (y/n): y
* [============================================================> ]1 / 1server1 status=running
---- service summary ----
Nodes: 1 / 1 Statuses: started=1 Elapsed Time: 0.12 s
The “Agent Puppet” module allows you to control the puppetd daemon and launch puppet client runs. The problem with this one is that it’s not up to date, doesn’t support tags and the noop option. Fortunately, some people have looked into it, so we’ll install this version while waiting for the official version to come out.
> mco shell '/etc/init.d/postfix restart'Do you really want to send this command unfiltered? (y/n): y
Determining the amount of hosts matching filter for2 seconds .... 1 * [============================================================> ]1 / 1[ch-bmf-srv-poc-5]exit=0: Shutting down postfix: [ OK ]Starting postfix: [ OK ] ...
If you encounter this type of error message in Puppet Dashboard when launching Puppet runs from Mcollective, you need to work on the puppet manifest, to comment this line:
1
2
3
4
5
6
7
8
9
[...]service {
'puppet-srv' :
name=>'puppet',
# Let this line commented if you're using Puppet Dashboard#ensure => stopped,enable=>false }
[...]
If you have this kind of message when starting MCollective:
/usr/lib/ruby/1.8/mcollective/pluginmanager.rb:169:in `load': no such file to load -- mcollective/facts/facter_facts.rb (LoadError)
from /usr/lib/ruby/1.8/mcollective/pluginmanager.rb:169:in `loadclass'
from /usr/lib/ruby/1.8/mcollective/config.rb:137:in `loadconfig'
from /usr/sbin/mcollectived:29
It’s because the path of the libraries is wrong1. Fix this in the MCollective server configuration file:
