PuppetDB1 allows you to retrieve data collected by Puppet such as facts and to use exported resources among other things. This data can then be used by other programs, such as the dashboard, or your own tools through an API. You can install the PuppetDB server on your PuppetMaster or on a separate server.2
Today it is possible to use 2 backends to store this data:
HSQLDB: in-memory database, with quite a few limitations including the 100 nodes maximum, but extremely fast (as it’s loaded in RAM)
PostgreSQL: classic database, with less performance (on disk), but with more flexibility and a greater possibility of extension over time (more than 100 Puppet nodes).
We will proceed with the PostgreSQL-based solution.
If you want more information, check out the different bottlenecks. In summary: if you have more than 100 clients, you’ll need a PostgreSQL database, increase your JVM, and increase the number of CPUs/cores.
[database]# For the embedded DB: org.hsqldb.jdbcDriver# For PostgreSQL: org.postgresql.Driver# Defaults to embedded DBclassname=org.postgresql.Driver# For the embedded DB: hsqldb# For PostgreSQL: postgresql# Defaults to embedded DBsubprotocol=postgresql# For the embedded DB: file:/path/to/database;hsqldb.tx=mvcc;sql.syntax_pgs=true# For PostgreSQL: //host:port/databaseName# Defaults to embedded DB located in <vardir>/dbsubname=//localhost:5432/puppetdb# Connect as a specific userusername=puppetdb# Use a specific passwordpassword=puppetdb# How often (in minutes) to compact the database# gc-interval = 60# Number of seconds before any SQL query is considered 'slow'; offending# queries will not be interrupted, but will be logged at the WARN log level.log-slow-statements=10
Here we define the database connection properties, as well as the credentials we created earlier.
Now let’s configure the number of threads:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
# See README.md for more thorough explanations of each section and# option.[global]# Store mq/db data in a custom directoryvardir=/var/lib/puppetdb# Use an external log4j config filelogging-config=/etc/puppetdb/conf.d/../log4j.properties# Maximum number of results that a resource query may returnresource-query-limit=20000[command-processing]# How many command-processing threads to use, defaults to (CPUs / 2)threads=4
Adjust the number of threads to your processor count divided by 2.
Then, we tackle the Jetty configuration:
1
2
3
4
5
6
7
8
9
10
11
12
13
[jetty]# Hostname to list for clear-text HTTP. Default is localhosthost=0.0.0.0# Port to listen on for clear-text HTTP.port=8080ssl-host=0.0.0.0ssl-port=8081keystore=/etc/puppetdb/ssl/keystore.jkstruststore=/etc/puppetdb/ssl/truststore.jkskey-password=CoaRwY6IL8KQd8H6SfZ7O9hHCtrust-password=CoaRwY6IL8KQd8H6SfZ7O9hHC
For the host, add the interface that will listen on ports 8080 and 8081. This notably allows the dashboard to connect to it.
warning
If possible and as a security measure, leave everything on localhost. Obviously, Puppet Master must be on this same machine if host and host_ssl are set to localhost
Then restart PuppetDB:
1
/etc/init.d/puppetdb restart
After a few seconds/minutes, you should be able to connect to port 8081 (ssl) or 8080 (non-ssl) (http://:8080|https://:8081), where you’ll have access to a nice interface:
[main]logdir=/var/log/puppetvardir=/var/lib/puppetssldir=/var/lib/puppet/sslrundir=/var/run/puppetfactpath=$vardir/lib/factertemplatedir=$confdir/templatespluginsync=true[master]# These are needed when the puppetmaster is run by passenger# and can safely be removed if webrick is used.ssl_client_header=SSL_CLIENT_S_DNssl_client_verify_header=SSL_CLIENT_VERIFYstoreconfigs=truestoreconfigs_backend=puppetdbreport=true[agent]server=puppet-prd.deimos.fr
warning
Remove the thin_storeconfigs and async_storeconfigs lines if you are using them, or set them to False
Then we’ll set up a file for Puppet’s configuration to tell it how to connect to PuppetDB:
1
2
3
[main]server=puppet-prd.deimos.frport=8081
And finally, a file to define the location of facts:
1
2
3
4
5
---master:
facts:
terminus: puppetdbcache: yaml
That’s it, your Puppet server now has a working PuppetDB backend! :-)
I have OutOfMemoryError errors in my logs and PuppetDB responds slowly link
To confirm that the problem is indeed due to insufficient memory, check that the file /var/log/puppetdb/puppetdb-oom.hprof exists, and make sure the content mentions OOM.
You’ll need to increase the Java Heap size (Xmx value) of your PuppetDB which requires more RAM. Increase this value:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
############################################ Init settings for puppetdb############################################ Location of your Java binary (version 6 or higher)JAVA_BIN="/usr/bin/java"# Modify this if you'd like to change the memory allocation, enable JMX, etcJAVA_ARGS="-Xmx192m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/log/puppetdb/puppetdb-oom.hprof "# These normally shouldn't need to be edited if using OS packagesUSER="puppetdb"INSTALL_DIR="/usr/share/puppetdb"CONFIG="/etc/puppetdb/conf.d"
For an idea of the value to set (where n represents the number of nodes):