Troubleshooting Ansible playbooks

the worst but honestly I'm super impatient so I usually leave that off but I'm going to be developing so I'm going to be doing a lot of iterations really quickly so it makes sense for me to just go ahead and update revision on launch I mean I I could not do it but I would have to come into the project section every time and click this sync button every time I made an update so this just kind of speeds up the process a little bit for me so I've got that in there I've already got an inventory I'm just going to use my generic one

I've already got credentials uh I'm going to now create a job template really fast I'm say add job template and I'll call this Playbook troubleshooting one because I plan on doing maybe a series of these so why not start with one this repository is where I'm going to keep all that good stuff so my inventory typ type in for my generic inventory I'll select that project it should be the troubleshooting one we just created Playbook troubleshooting there it is I'm going to select a playbook in here and it's going to be Playbook one- Greg right because this is the one I'm actually making changes to

I'm going to leave the Playbook one in its pristine broken State it's weird to call it pristine but in its existing state I'm going to leave it there credentials I should be able to use uh let's say for example I'll use my demo credentials in there that should be everything I need so I should be able to click save and let's launch and see what happens now throughout the course of this I don't want you to have to sit and be bored while I do all these things so uh I'm going to do a little bit of editing to cut out the weight time in

between these runs all right so this says the job has failed and there's no output but I bet you if I click details and then back over to Output I do indeed have some output so we're unable to read either as Json nor yaml so normally when you see that message that means you've really messed up the file uh to such an extent that it can't really read right so it's saying line one column one let's take a look at line one column one here line one column now typically a yaml file is going to start with three dashes so in this one and and

watch the syntax highlighting on this HL file so VSS code uh which is Visual Studio code uh it's free uh IDE from Microsoft you get in there and it recognizes ammo file and so it'll do syntax highlighting so when I actually fix this from four dashes to a emo file should start with three dashes you should see the syntax highlighting change everything looks very different so I'm going to save that and then I will commit to my repository so that was line one column one commit now I should come back over here and if I want to rerun the job so when you're in the

job template section and you fire off a job it immediately takes you over to the job section and just shows you the output of that job and if I'm in here I click the rocket ship and I can say all so run against or rerun using all the exact same settings against all the same host so if I had say maybe extra variables that I passed or something like that it'll use those exact same ones and rerun it again so it's just kind of a little time save so we've got our failed output again we're going to see this a lot we're going to see

a lot of red before we actually see the green so click details and then output again H line one air one now let's see what else we have here fending line line 2 column 7 maybe elsewhere in our Playbook job for learning troubleshooting here let's take a look oh common error right this is actually user error so I did commit the change but I forgot to synchronize the change right committing the change just saves it in the local uh repository on my machine sync means actually push that up to my repository so now that I have synchronized it let's rerun so essentially I just reran

the exact same Playbook again all right we have our failed again detailed output let's see did we see something different we did right so now it says there appears to be uh did not find expected key appears to be line 11 column 5 and it's showing us here and display variable info so if I pop over here line 11 that's right here now notice the syntax highlighting here under task see where it says uh name right and then the register and they're both kind of this I don't know what is that kind of a light purple sort of color and over here everything's blue that's

because we have our indentation wrong so I could do one of two things I could either grab this task and I can move it to the left or I can grab this task and move it to the right so I tell you what I'm going to do is I'm going to highlight all three of those and I'm going to hit control and the right bracket and it's going to move all that to the right now you see as soon as I did that the syntax highlighting changed for these right here right so we did a little bit of a fix there I'm going to contrl

S to save that I'll put some comment in here I'll wait for it to commit then I will synchronize those changes now let me come over here and rerun the job and let's see what we get next time all right failed no output details output let's see what we got uh conflicting action statements blah blah blah the error appears to be line 11 column 7 but it may be elsewhere depending on the syntax so conflicting action statements so it's saying conflicting between debug and message so action statements action that's generally going to be what it's referring to as a task like I don't know which task to actually run here so if you take a look at these two columns you see I have an.

built-in. debug which is the module I'm trying to call and the message is a parameter for the module and you see I accidentally have them on the same line so what I need to do is I need to grab this line I need to shift it to the right one position which is really going to be two spaces in yo files generally kind of the good practice I'm not going to say best practice but good practice is going to be two spaces right so it's going to be indented in so if a parameter uh is specific to a module it's going to be indented into

the right of it right so that's really saying hey this module has precedence jump over a little bit and then here are all the parameters associated with it so now we've got that situated it's not going to be confused on which one of these is actually the uh the module I'm trying to call so let me commit that let me sync my changes and I'll come back over here fire it off again and let's see what we end up with all right another failed output let's click on details go back to Output again what do we have this time line 11 column 7 says couldn't resolve module SL action so I immediately see what it is uh I am bad at grammar sometimes and so an.

built-in. DG instead of debug right so if it ever says hey I can't find a module name nine times out of 10 I've just misspelled it uh the other 10% of time it's because I'm using an execution environment that doesn't have that collection in there or if you're using just standard command line anible maybe you don't have that collection installed so double check first did I actually spell The Collection name or the module name right and uh is the collection installed so this one is built in which means yes it is there we just have to make sure it's spelled correctly so an. built-in.

debug builtin I misspell that one a lot too for whatever reason so save that commit sync and now I will fire this off yet again again all right so let's take a look at the output there was some success so at least it didn't error out when it ran this time but it says uh could not match supplied host pattern ignoring test host one that means in my inventory I probably don't have a test host one now here's a little trick so I'm going to open a new tab on the inventory right there so I can pull it up I'll find my generic inventory so

sometimes you can go into the host section or I could be calling a group right so if I'm calling a group I tend to run to fewer errors so if you can put your things in groups and just put the group name at the top of your uh playbook in the host section often times you'll kind of remove some errors there but in this instance I'm calling an individual host well test host one I think is what it was called right yeah test host one that doesn't exist in here but I have run into before where a host does 100% exist in here but it keeps giving me that hey I'm not matching anything I'm not finding anything well if you go in your inventory check over here whether it's enabled or not so even if a host is in here so say for example this host 44.

example.com if I call it but it's turned off then a Playbook isn't going to be able to follow it essentially that invalidates it in the inventory it it's essentially a comment at that point so your playbook will never actually find it so if you 100% know that thing is there just check to make sure that it's actually enabled so i' I've seen that before where something came in disabled for whatever reason so in this instance I could go with a group but you know what I'm just going to run it against this Greg Rocky 9 host I'm going to copy that name and I'm going

to put that in my playbook so right here I will put it in the Playbook I could also change the host section to all and then set the limit option in the uh job template to be uh Greg Rocky 9 so uh the limit section really allows you to narrow down even more specifically what you want to operate against so it's kind of a little fudge there so if all of your playbooks you just wanted to have them run against all and then you use the limit field you could do that as well so I'm going to go ahead and save that put my commit

in my sync I will launch this and let's see what it comes up with this time ah now we've got a failure this time so I'm going to go back to my output so it was able to actually find it but now it's saying fail to connect to the host via SSH so I'm getting permission denied so I'm not able to actually log in so let me double check my credentials again so if I go to the details section I can just click on the job template name there and it takes me right in I will click on edit sure enough I accidentally use the

demo credentials what a dumb dumb I will use my G credentials and I will save that job template and I'll give it a launch all right we've got another error what do we have here invalid variable name in register specified for sudoers collect so what makes the variable name invalid sudoers D collect so I'm going to take a look at that one right here so they're referring to this section right here uh for the collect sudoers file so really what I'm trying to do is I'm trying to just do a shell command which means I'm going to just fire something off of off as if

I'm right on the shell like I'm ssh in I'm just going to issue a command I'm going to cat out the sudoers file well the register command says hey save all this output to a variable and name it sudoers D collect the only problem here is uh anible doesn't allow you to do dashes in variable names kind of the only special symbol you can do is an underscore so I'm going to go ahead and change that to an underscore that was something that uh I ran into problems with early on myself all right let me fire this off and let's see what the results are

we've got an another err let's take a look at it so I was able to authenticate the uh variable name is now correct in the task but I am getting change true H permission denied on the Su doers file right cat Etc Su doers that's because I am just a standard user in there and I need to actually escalate privileges to access this so I need to use the Pudu command well in asemble in a Playbook the way you actually tell it to escalate privileges is I can use the become option so here I can say become true and what that will essentially do is

for that credential it will look at its privilege escalation so if it's a Windows box on that privilege escalation I would say like run as administrator in here I'm doing sudu so it will do s o before my command and it will operate in that book fashion now you can see I've added it to just the individual task right here and so there's a concept of least privilege access right I only want to elevate those privileges when I absolutely need to and in this Playbook that's the only command that really needs privilege escalation so a lot of times my playbooks a whole bunch of commands

require that privilege escalation and what I can do is I can uh up here at the top I can put become colon true at the top and that means it's going to be inherited by all the the task associated with so everything will be running as sudu right it's going to do escalated Privileges and in that's the case if a whole bunch of things need it I can also do on individual tasks that don't need it say I have 10 that do and uh one that doesn't on that one that doesn't I could say become colon false and it'll just turn it off for that

one individual right so there's a little bit more efficiency in that I think a lot of people prefer to do become true on the individual ones they want because it's kind of easier to remember and you don't accidentally throw extra commit commands in there and have them automatically assume those additional privileges so we have added that change we've committed it let's fire it off and see if it's actually able to parse that file now all right so it's saying successful but let's take a look at the output we have run it it has successfully escalated privileges it's gathered all the information and it is displaying

it to screen so it looks like we have reached success so at this point you can say hey you know what this Playbook actually ran and it gave me all the information I wanted um I'm not quite going to stop there I'm going to change two more things um these are not necessarily requirements these are just like pet peeves of mine so whenever I'm doing collection of information on a shell command uh the shell command always assumes there's a change right so if I take a look at uh the output here it's you know it's going to show me everything that I had pulled out

of that file but in here it says changed true so it assumes since I ran that shell Comm in and then things came back I actually made a change whereas here I'm really just gathering information so what I'm going to do in my playbook is I can add a task level parameter that applies to the task itself and I'm going to say changed when false so essentially here I'm just saying it's never going to show changed right so for this command never show changed on it right and that's going to kind of speak to the item potent nature the item potency means um you can

run a command over and over it's only going to make make an actual change on the system when it needs to well if it says change when it didn't actually need to that's going to give kind of some false positive information so I usually like to change those if it really is just information gathering there is absolutely no chance it can make a change go and just adjust that change when condition now I could add additional parameters in there because it is a conditional like any other so I could look at the registered output I can look at the contents and look for something very

specific inside uh a condition which would you know indicate to me oh I actually did make a change on this um in this case it never will so I'm just going to say change when false the other thing I'm going to do is on the debug module here it's got a couple of options that you can actually use so I'm going to highlight all of that I'm going to hit control for slash which comments it out I'm going to paste in a copy right there and you can either do MSG for message and it allows you as you can see here I just crafted a

regular text message um or I can do something like VAR and so VAR will allow me to specify a specific variable and it will spit out the contents of said variable now if you take a look this right here I am getting rid of the double quotes and the uh double curly braces and really in this above output this is another kind of gotcha so while technically this command ran that format is bad it really should have been the double quotes around the entirety of the message and then any extra variables I need in there you just put the dble cly braces you don't actually

have to do anything else with the uh double quotes but I digress down here if you're actually using the VAR option you don't have to use any curly braces you don't have to use any quotes it'll actually just spit out the contents of a variable so here I'm doing sudoers collect uh. STD outlines and so that's going to spit out that very specific portion of it the STD lines portions can all get out if I got rid of STD outlines it would really just show me every bit of the output so it'll show me the change condition it'll show me St out STD Out Lines

all that good stuff but here I really just want to see lines so I am going to commit synchronize that and let's fire it off and then make sure I didn't mess anything up but also let's see how the output has changed all right success let's take a look at the output so again remember in the previous run it actually showed change if I wanted to see that I could go to the job section and right here I could look at the old Playbook run right so it showed change and all this stuff was jumbled up instead I want to see the output of the

newest and I'm looking for Playbook run right there troubleshooting one so again it's okay green instead of orange changed or slightly yellow I guess depends on your monitor right so it's really just showing hey I collected the information but I didn't actually make any changes and then the VAR output see how this is so much more readable now it uh truncates this output but if I go to the Json section it'll actually show it all to me right here right so I can scroll through the entirety of the file and it's a lot easier to look up with the the vars formatting now I could

format it with some special characters in the message section but you know why go through all that work when I can just throw VAR at it it'll actually show me all that output so at this point uh if you have done it on your own uh I hope you uh learned a couple of things if you watched along hopefully this will help you kind of figure out how to match an air message to hey where do I find this to troubleshoot maybe show you some caveats maybe some gotas that you run into already uh I would at this point like to say thank you for

watching if you have any questions or comments or you would change this or adjust this or you want to see me do something different please reach out I genuinely enjoy feedback and I'm usually pretty pretty talkative I I like meeting new people and I like it uh when people are interested in automation if you have the want or need for somebody to come in and help you do some of this automation please feel free to reach out to us we love helping folks I love doing this stuff I love the education piece of it absolutely enjoy that we also can help you build out your

automation practice we can help you build out your clusters all your infrastructure let us know we're happy to help if nothing else happy automating happy troubleshooting and we'll see you next time bye