.

Puppet & Splunk FTW

Statler_and_Waldorf

Puppet recently announced the release of a brand new app called “Puppet Enterprise App for Splunk”:
https://splunkbase.splunk.com/app/3100

The app is currently certified for Splunk >= 6.3.0, and I successfully deployed it on Splunk 6.4.0.

I would like to share my journey of deploying the Puppet Enterprise App for Splunk in conjunction with the deployment of a multi-node Splunk cluster using the official “Puppet Module for Splunk”:
https://forge.puppet.com/puppetlabs/splunk

The module is currently certified for Puppet >= 4.0.0 < 5.0.0 on CentOS, and I had no issue using it with Puppet Enterprise 2016.1.1 on Ubuntu 14.04.

I have been deploying Splunk Clusters with Puppet across a diverse range of customers over the last 4 years, and I have been using a heavily customized version of the Puppet module released by Harvard University:
https://github.com/huit/puppet-splunk

I thought that I would give the official Puppet Module for Splunk a crack, and I am happy to say that it has some great features!

To really put it through its paces, I used the official Puppet module to deploy a 3 node Splunk index cluster, a 3 node search head cluster, a master/deployer node, a deployment server, and a Sensu monitoring server in AWS.

To top it all off, I deployed the “Puppet Enterprise App for Splunk” and I will cover off some of its features later in this article.

octoninja

The Puppet module for Splunk is quite straight forward to use, and it includes the ability to deploy the server version of Splunk as well as the Universal Forwarder – which is used to forward logs from client servers to Splunk. The module also has a comprehensive list of custom types to manage Splunk’s many and varied configuration files, including the all important server.conf & web.conf files.

The Puppet module documentation doesn’t go into detail for configuring the various types of roles for Splunk servers, however I am happy to share our sample configs here 🙂

We are very security conscious at Katana1 so I added a section to enable https/ssl for the Splunk WebUI:
init.pp:

$web_ssl = '1',

  splunk_web { 'splunk_server_web_ssl':
    section => 'settings',
    setting => 'enableSplunkWebSSL',
    value => $web_ssl,
    tag => 'splunk_server'
  }

}

I added a class to manage Splunk servers, including a splunk_server type for Splunk license clients:
server.pp:

class splunk::server {

  class { '::splunk::params':
    version => '6.4.0',
    build => 'f2c836328108',
  }

  include ::splunk

  # Splunk License configuration
  if $splunk_server_type != "lm" {
   splunk_server { 'splunk_license_master':
     section => 'license',
     setting => 'master_uri',
     value => 'https://$splunk_lm:8089',
     tag => 'splunk_server'
   }
  }

}

I added a class to manage Universal Forwarders, including a default deployment client configuration:
uf.pp:

class splunk::uf {

  class { '::splunk::params':
    version => '6.4.0',
    build => 'f2c836328108',
  }

  include ::splunk::forwarder

  # Deployment Client configuration
  file { "/opt/splunkforwarder/etc/apps/uf_dc":
    mode => '0755',
    owner => 'splunk',
    group => 'splunk',
    ensure => directory,
    recurse => true,
    source => [
      "puppet:///modules/splunk/apps/uf_dc",
    ],
    require => Package['splunkforwarder'],
  }

}

I added a class to manage the Master Node, including configuration for the Deployer role:
mn.pp:

class splunk::mn {

  # Master Node configuration
  splunk_server { 'splunk_master_node_mode':
    section => 'clustering',
    setting => 'mode',
    value => 'master',
    tag => 'splunk_server'
  }

  splunk_server { 'splunk_master_node_replication_factor':
    section => 'clustering',
    setting => 'replication_factor',
    value => '3',
    tag => 'splunk_server'
  }

  splunk_server { 'splunk_master_node_search_factor':
    section => 'clustering',
    setting => 'search_factor',
    value => '2',
    tag => 'splunk_server'
  }

  splunk_server { 'splunk_master_node_pass4SymmKey':
    section => 'clustering',
    setting => 'pass4SymmKey',
    value => '<passkey_goes_here>',
    tag => 'splunk_server'
  }

  splunk_server { 'splunk_master_node_cluster_label':
    section => 'clustering',
    setting => 'cluster_label',
    value => 'cluster1',
    tag => 'splunk_server'
  }

  # Deployer configuration - SHC
  splunk_server { 'splunk_deployer_shclustering_pass':
    section => 'shclustering',
    setting => 'pass4SymmKey',
    value => '<passkey_goes_here>',
    tag => 'splunk_server'
  }

  splunk_server { 'splunk_deployer_shclustering_label':
    section => 'shclustering',
    setting => 'shcluster_label',
    value => 'shcluster1',
    tag => 'splunk_server'
  }

  # Deployment Client configuration
  file { "/opt/splunk/etc/apps/mn_dc":
    mode => '0755',
    owner => 'splunk',
    group => 'splunk',
    ensure => directory,
    recurse => true,
    source => [
      "puppet:///modules/splunk/apps/mn_dc",
    ],
    require => Package['splunk'],
  }

}

I also added a class to manage the Peer Nodes, there was an issue adding an empty stanza for the replication_port section so I added a custom exec resource instead.
pn.pp:

class splunk::pn {

  # Peer Node configuration
  exec { 'splunk_peer_node_replication_port':
    cwd => "/var/tmp",
    command => "sed -i '/^stack_id = free/a [replication_port://9887]' /opt/splunk/etc/system/local/server.conf",
    unless => "grep replication_port /opt/splunk/etc/system/local/server.conf 2>/dev/null",
    path => ["/usr/bin", "/usr/sbin", "/bin"],
  }

  splunk_server { 'splunk_peer_node_master_uri':
    section => 'clustering',
    setting => 'master_uri',
    value => 'https://$splunk_mn:8089',
    tag => 'splunk_server'
  }

  splunk_server { 'splunk_peer_node_mode':
    section => 'clustering',
    setting => 'mode',
    value => 'slave',
    tag => 'splunk_server'
  }

  splunk_server { 'splunk_peer_node_pass4SymmKey':
    section => 'clustering',
    setting => 'pass4SymmKey',
    value => '<passkey_goes_here>',
    tag => 'splunk_server'
  }

}

Watch this space for additional configs to run Splunk as a non-root user, as well as additional classes for Search Head Clusters, Heavy Forwarders, Data Collection Nodes, Deployment Servers, and License Masters.

After deploying the 8 node Splunk cluster with the official Puppet module, I also deployed Sensu using the official Puppet module:
https://forge.puppet.com/sensu/sensu

Sensu is an awesome open source monitoring framework built for the cloud. We use Sensu extensively at Katana1 to monitor Splunk clusters for our customers.

Once the Index cluster and Search Head Cluster were running, it was time to deploy the Puppet Enterprise App for Splunk!

splunk

I added the “puppet-enterprise” index to the add-on for the Peer Nodes, then I created a specific add-on for the Universal Forwarder on the Puppet Enterprise server, and finally I pushed the Puppet Enterprise App for Splunk to the Search Head Cluster.

The app provides valuable insight into the health of Puppet Enterprise, both for monolithic or split installations.

The initial version of the app provides 6 data models and 4 dashboards.

Dashboards:

  • Console Services Overview
  • puppetdb HTTP Metrics
  • puppetserver Compilation Metrics
  • puppetserver HTTP Request Metrics

pesa1

pesa2

There is a detailed explanation of the features of each dashboard on splunkbase:
https://splunkbase.splunk.com/app/3100/#/documentation

Pro Tip: Accelerate the data models for faster dashboards:
https://splunk/en-GB/manager/pe-splunk-app/data_model_manager
Click “Edit” -> “Edit Acceleration” -> Tick the checkbox next to “Accelerate” -> Select a “Summary Range” -> then click “Save”

Future releases of the app may include Puppet specific data like resource events, reports, and data from the various metrics endpoints for PE services.

There is another app available on splunkbase called “Splunk App for Puppet”:
https://splunkbase.splunk.com/app/1701

The Splunk App for Puppet is quite dated, however it inspired another app called “Puppet Pulse” by Wil Cooley:
https://github.com/wcooley/splunk-puppet

Puppet Pulse has some cool dashboards and I am working on incorporating them into the Puppet Enterprise App for Splunk. I will write up another blog post in the near future with my progress.

Please feel free to post comments below, or email/tweet me with any questions or queries.

Happy Splunking & Puppeteering 🙂

Luke Harris
Data Analytics Practice Lead
luke@katana1.com
twitter.com/skywalka

Add your comments

Your email address will not be published. Required fields are marked *