Install Performance Co-Pilot via Ansible and Ascender
Performance Co-Pilot (PCP) collects and stores performance metrics on a server so an admin can look back in time when troubleshooting. This demo opens with the nomenclature: a collection host gathers stats on itself, while a monitoring host reaches out and pulls metrics from many collection hosts so there is one place to look instead of SSHing into every server. Greg Sowell explains why pulling from the monitor is easier on the collection hosts and notes the documented rule of thumb of roughly a thousand collection hosts per monitoring host.
From there the video moves into Ascender, touring inventories, projects, credentials and job templates before launching a collector playbook and then a monitor playbook, both kept in public Git repositories. The collector playbook installs PCP, applies a Jinja2 template that restricts which remote subnets may connect, opens TCP port 44321 in the firewall, configures SELinux, and enables the service with a handler that restarts it when configuration changes.
Anyone managing fleets of Linux servers walks away with a repeatable, idempotent way to roll out PCP collectors and a central monitoring host, and sees the per-host log directories filling on the monitor as soon as the run completes.
Key takeaways
- A collection host gathers PCP metrics on itself; a monitoring host pulls those metrics from many collection hosts into one place.
- Pulling from the monitoring host is lighter on collection hosts than pushing, so it skews the collected performance data less.
- Documentation suggests keeping a monitoring host to roughly a thousand collection hosts, though a beefier server can take more.
- The collector playbook installs PCP, templates the config with allowed remote subnets, opens TCP port 44321, and sets SELinux and firewall rules.
- Ascender job output is idempotent, showing changed items on a fresh server and ok items when nothing needs adjusting.
- The monitor playbook writes a per-host config file and directory, then starts PCP so it begins pulling metrics immediately.
Questions this video answers
What is the difference between a PCP collection host and a monitoring host?
A collection host is any server running Performance Co-Pilot to gather metrics on itself. A monitoring host is configured to reach out and pull metrics from many collection hosts, giving an admin a single place to SSH into rather than connecting to each server individually.
How does the Ansible playbook configure PCP on collection hosts?
It installs the PCP packages and system tools, uses a Jinja2 template to write the PCP configuration with the remote subnets allowed to connect, sets the listening address on TCP port 44321, configures the firewall and SELinux, then starts and enables the service with a handler to restart it after changes.
Should PCP data be kept on each host or on a central monitor?
Greg recommends both. Keeping data locally means it survives if the monitoring host goes offline, while the central host gives one place to look. He also suggests exporting metrics to a monitoring platform for long-term graphs and analysis, since the approaches are not mutually exclusive.
About this video
Installing Performance Co-Pilot (PCP) on collection and monitoring hosts can be a little obtuse at first. Greg explains how it works, then demonstrates making the changes via automation with Ascender!
This video is part of the Ascender Pro playlist. Browse every CIQ video by product and topic.
Transcript
hey everybody I'm Greg Sowell and welcome to another ciq demo today I'm going to be showing you how to use the Ascender automation system here with your anible playbooks to configure PCP or performance co-pilot so first what is performance co-pilot it is a set of applications tools if you will to collect performative information on your host right on your servers and so what's this good for it's good for tweaking and tuning and making your system more efficient but I think most people are using it for troubleshooting performance issues so that means I'm collecting information right because it actually collects and stores for whatever amount
of time you determine it should um and then if you have a performance issue you can kind of go back in time and see what happen right like see what's going on inside your environment so really good for that I uh am going to be using Ascender here to fire that stuff off but I thought maybe I would give you uh some quick uh nomenclature right so kind of I'm going to be talking about a couple of different concepts and I just want to make sure it makes sense to you so in PCP you have something called a collection host and the collection host
is really just any server that's collecting metrics on itself right that is the collection host so anything running just PCP collecting stats on itself collection hosts now if I have an admin like in my environment and they want to actually pull that information generally what they're going to do is SSH into that server and they're going to be able to use the PCP tools to look at all that information that's been gathered right that's kind of your your standard form that you're going to see this move in now imagine you have in this example six servers well you're going to be connecting to each one
of those individually to pull that information right that could get a little tedious right maybe if we have six no big deal but what if we have a thousand or 6,000 right it gets a little bit cumbersome to kind of try and manage that so what you can do is install another another set of nomenclature here a monitoring host right so collection host is just a server that's collecting information on itself a monitoring host is a server that is designed to collect information from all these collection hosts right it's pulling all that information in let say pull you have a couple of options there you
View full transcriptHide full transcript
can do a push from the host or you can do a pull from the server right the monitoring host can reach out and pull there's supposedly a couple of um thoughts on this now the collection host can push but it's going to be a little bit more resource intensive which can skew some of your information right skew some of your uh your uh performance information that you're collecting whereas the monitoring host can reach out and pull that information and so it's easier on your collection host right it's going to impact your performance monitoring statistics less um it is more impactful to the monitoring host but
who cares as long as he grabs that information and goes you should be good in some of the documentation I've reading it says don't uh don't necessarily load up your monitoring host Beyond about a thousand collection hosts I guess that's kind of their their general rule of thumb I'm little bet you can kind of beef that server up a little bit and accept more but in my environment I am pulling like and and this automation will actually be configuring all this I'm going to configure a collection host and a monitoring host here moment um and again that becomes your One-Stop shop right so the admin
instead of sshing in those individual servers can just SSH into the monitoring host and gain access to all that information now having said all of this uh I think it's cool to collect that information keep it locally right for troubleshooting purposes but also you can export this off to monitoring servers right so the statistical information um can allow you to jump in and right this really more or less what it does is at least from what I've seen PCP allows you to run command line tools uh that emulate regular commands right so you can um on a server I could run top to see kind
of what applications are running right now what their resources are or uh using PCP I can run like PCP rtop I believe something like that right so it gives me kind of an emulated experience of the existing tools that I'm a custom to now I can do that that's great on the spot troubleshooting but I'm a big proponent of using monitoring system so if you could export this information into a monitoring platform collect that for long-term statistical analysis look at graphs things like that I think that could actually be useful as well so these aren't necessarily mutually exclusive I think you could probably do both
things but keep all that mind so having said all of that let's jump back over to Ascender so I'm going to go to my job template section which is where I uh have all of my automations kind of created well the quick and dirty we have inventories that's going to be all of the rather it's list of hosts of all the things I could operate against and I'm just using my demo generic inventory right here where I've got a couple of hosts right we got come on host load and then I've got rocky n Rocky 9-2 and you can see there are in uh Associated
groups right so the PCP host groups those are going to be the ones where I'm configuring performance co-pilot in the collection right there collection host and then PCP monitor that's going to be my monitoring host obviously in your real environment you're going to have more than two machines going but this is just for demo purposes next I've got projects this is how I pull in my git repository so all of my playbooks and they are in public G repost so you can grab all this and utilize it yourself or make uh modifications to it whatever you please but projects is how we pull those um
playbooks those templates all those resources they get pulled in into a project credentials how to do I actually log into various machines it's going to store your uh SSH credentials certificates all that good stuff to be able to connect in and job templates are how we kind of bring it all together we tie all those things together so I'm going to look for PCP performance copilot stuff so I have my collector install and configure and my monitor so we're going to start with the collector going go ahead and click the rocket ship there this is going to fire off and through the magic of editing
you don't have to watch any of this I'll see you in just a second all right so now that that's complete I'm going to scroll through the output if you've never seen job output this is kind of an indication of what it looks like if you say uh say for example you see green it just says okay this is item potent which means I can rerun this automation again and again and if it needs to make a change it will if it doesn't it won't in this case right all this was brand new this is a clean server so you can see all these change
entries this means I actually did make an adjustment right I did make a manipulation here here and so I can on any of this information I can it's installing PCP packages I can click on it go to the output and I can see exactly what transpired there right it's really cool for auditing purposes I can go back in time and see all this stuff but let's take a look at the host itself so I'm going to tab over so here is my host and I should be able to run just the PCP command and it should now show me yeah I've got performance co-pilot installed
here um something else I'm configuring on these playbooks and I can show you here momentarily is that I'm setting it up so that it listens on TCP Port 44321 uh for the monitoring host to actually be able to reach in and extract that information so let me double check that that actually got configured hit enter and I do see it listening on my port so as far as PCP goes I'm done right pretty hands off pretty easy all I had to do was run the automation it connects into all those host po configure PCP exactly the way I want sets them up to listen and
now we're good so as promised I'll show you a little bit of the Playbook this one is PCP install and in here you can see I'm setting up a few set of variables right up here at the top enable services this is because I'm going to go through and enable some various things and on this specific one I'm enabling those Services I'm just kind of doing variable IED with loops up here at the top cuz I knew I was going to copy and paste this and use it for other things and I would need to turn certain things on and off so it's just I
like keeping all those options up the top next you're going to see remote subnets now this is what are the IPS or ranges of the monitoring host that might potentially be connecting in to my uh PCP collector here to extract information right it's basically I'm putting an access list in here so you just fill out whatever subnets or whatever individual IP addresses those monitoring hosts are going to be coming from to secure those just a little bit better next you can see here I'm just doing the PCP install with the system tools next after that I am using a Ginger 2 template which are really
awesome they're basically text files that I can do variable replacement I could do loops inside of there I can do conditional information inside but most people it's just a text file with variables I'm going to replace I'll take it and I'll shove it onto the server so that's configuring the uh PCP process right that's the um the part that's got like the IP addresses for the various rang that are going to be allowed in stuff like that uh next I'm going to be figuring the IP address that this thing is going to listen on I'm setting up the firewall I'm configuring uh SE Linux to
allow all this stuff to one gather information but also allow things to come in and uh grab that info then I'm starting and enabling the service and then also have a Handler at the bottom so that if I rerun this and there were modifications it'll go ahead and restart the services so that they'll accept all the new information go back to my main screen here I'm going to pop back over to my Ascender I'm going to go into my template section and again I'm going to go PCP now I'm going to configure and install the monitoring server so again the magic of editing you're not
have to sit and watch this we'll be back when it's done all right so this completed successfully let's take a quick peek at the PCP monitoring configuration Playbook so the setup setup rather at the beginning is very similar and here in the task section you can see that I am installing the PCP packages required here and then after that I am creating the configuration files for each host so you could in the config file you could edit one config file and have all of your hosts in there and this U method I'm actually per host I'm just creating its own individual file which really is
just setting up um the IP of the host you're going to connect to the folder you're going to save it into and then kind of the interval that you're going to um I think pull this at as well as how long you're going to store this information right so all of that's kind of baked into a template file that is absolutely modifiable by you you feel free to get in there and get crazy with it after that I am going to Loop through and create all of the uh directories for the various hosts I'm going to start the PCP service and that's it like as
soon as you have the configs in there you start the service it will start reaching out and collecting all that information so let's pop into the server really quick and take a look so all of this I am configuring to save in the v log PCP PM logger and do a ll I can see my Greg rocky9 host is in there so I'm going to CD into Greg rocky9 do an LL ah here you are you see that it reached out and grabbed a bunch of information and is already storing it here on the server um it was configured pretty quick pretty succinctly um I
didn't have to make too many tweaks or Tunes now you do have the option of on the collection host being very specific right very granular on what you want them to to grab or not and it's super easy to just set up a template file that you can use to push all that information to your very service you can even maintain different template files for different types of servers right so it's entirely up to you super easy to modify all of this and push to your environment as well as have an individual monitoring host where you can pull that um I'm a big fan of
having the PCP information on the individual host as well as the monitoring host because if your monitoring host goes offline well you still have that information you can just grab it and get it from there but in an Ideal World you want to go to one place if you can for all of your information now this is the part where I say if you have any questions or comments I would love to hear them if you would tweak or tune this or change this in your environment or you'd modify the playbooks let me know about it I'd love to hear it um if you would
like help with any of this with automation with migrating from any other Enterprise Linux over to Rocky or standing up Cinder in automation we have become experts in all things migration and Linux and automation so please feel free to reach out to us and if nothing else bye
Built for scale. Chosen by the world’s best.
2.75M+
Rocky Linux instances
Being used world wide
90%
Of fortune 100 companies
Use CIQ supported technologies
250k
Avg. monthly downloads
Rocky Linux
9
Enterprise products
Spanning the kernel to the orchestrator
Have questions about your infrastructure?
Talk to a CIQ engineer about Rocky Linux, HPC, and AI infrastructure.