Thanks to Ido Raday for this amazing troubleshooting guide!
This is an interactive guide for debugging the auto triage process.
There are several requirements:
Access to the machine
Basic knowledge in shell commands
For any question please reach out to the server team
The Auto triage process is triggered by the corresponding issue.
Before we debug the auto triage we first need to make sure that a new issue created.
We also need to keep the alert_id for future steps.
Enter the issues pages and find the correct issue.
Click on the relevant issue
Enter the relevant issue → Overview
On the issue page and copy the alert_id from the address bar
If the issue was created - goto “Does the issue have Auto triage details?” section
If the issue was not created - goto “Verify the Auto triage process result” section
You can identify an issue with auto triage by the symbol next to the headline
If the Auto triage symbol appeared - Great! Close the document
If the Auto triage symbol did not appear - goto “Verify the Auto triage process result” section
If the issue was not created it can be due to several reasons:
There is already an active issue on the device
There is some issue with the data collection
There is some issue with the rule
Make sure to find the reason and only after fixing it continue to the next step
If the issue was created - goto “Does the issue have Auto triage details?” section
If you can’t manage to find the problem - goto “Open ticket for Server team” section
Every issue should trigger event in the Auto triage service.
The auto triage will run the corresponding playbook (if available) and will send the result back to the server.
This step will check the result the server got from the playbook,
Enter psql and run the next commands (replace ALERT_ID with the correct one):
\x select * from automation_job where alert_id = 'ALERT_ID'; |
Copy the job_id as you will need it for the next steps
If the query returned empty result - goto “Is Auto Triage feature enabled?”
If the query returned success result - goto “Auto triage process result Was a success”
If the query returned Not_Available result - goto “Auto triage process result Was a Not_Available”
If the query returned error code FILE_IS_MISSING - goto “Auto triage process failed with error code FILE_IS_MISSING”
If the query returned empty result UNABLE_TO_GET_DEVICE_DATA- goto “Auto triage process failed with error code Unable_To_Get_Device_Data”
If the query returned empty result INSUFFICIENT_DATA_ACQUIRED- goto “Auto triage process failed with error code INSUFFICIENT_DATA_ACQUIRED”
If the query returned empty result PLAYBOOK_RUN_FAILED- goto “Auto triage process failed with error code PLAYBOOK_RUN_FAILED”
It seems like the auto triage process did not run.
There are two ways to enable the Auto triage feature:
Server Application.conf (not recommended- this config is removed every install )
The path for the file is: /usr/share/indeni/conf/application.conf
The flag name is enable-automation-process
Using the following command you can check the flag status:
less /usr/share/indeni/conf/application.conf | grep enable-automation-process
Psql db
Enter the db by simply writing psql
Using the following command you can check the flag status:
select * from configuration where key ='automation.enabled';
If no line returned - the feature is disabled
If the feature is disabled in both ways - goto “Enable the feature flag”
If the feature is enabled at least in one way - goto “Open ticket for Server team”
Server Application.conf (not recommended)
Vi to /usr/share/indeni/conf/application.conf
Change the value next to enable-automation-process from false to true
Restart the server using imanage → 3
psql db
Enter the db by simply writing psql
Using the following commands you can check the flag status:
delete from configuration where key ='automation.enabled';
insert into configuration (key,value) values ('automation.enabled','true');
After creating a new alert - goto “Open ticket for Server team”
It seems like the auto triage process worked on the server-side.
If the issue does not show the alert it can be due to UI issue, please open a ticket to application team
It seems like the auto triage process did not find a playbook to run for the issue.
Search the triage process log for alert_id for example:
less /usr/share/indeni-services/logs/automation.log | grep 0c7968cd-8df9-4272-852a-ebd3bea2b130 -A20
Find the log block the related to the alert_id, for example:
2019-10-05 19:53:38,263 - INFO - automation_registration.py - New automation request, alert_id: 0c7968cd-8df9-4272-852a-ebd3bea2b130, device_id: 270d7888-ede5-419d-b968-ab45c8a08c07, rule_name: DeviceMonitoringSuspended, vendor_name: paloaltonetworks 2019-10-05 19:53:38,264 - INFO - playbook_catalog.py - Get playbook for rule: DeviceMonitoringSuspended vendor: paloaltonetworks 2019-10-05 19:53:38,264 - INFO - playbook_catalog.py - playbook for rule: DeviceMonitoringSuspended vendor: paloaltonetworks is None 2019-10-05 19:53:38,264 - INFO - automation_registration.py - New job created, job_id: d8dad379-c130-4135-9c1a-35ce52fd201d, alert_id: 0c7968cd-8df9-4272-852a-ebd3bea2b130, device_id: 270d7888-ede5-419d-b968-ab45c8a08c07, playbook_file: None |
If the data does not as expected - goto “Open ticket for Server team”
The error FILE_IS_MISSING indicated that the playbook file is missing.
Open the catalog:
less /usr/share/indeni-knowledge/stable/automation/playbooks/playbook_catalog.yaml
Find the playbook that matchs the issue.
Make sure the playbook file exists
If the data is now displayed as expected - goto “Open ticket for Server team”
The error Unable_To_Get_Device_Data indicated that the Auto triage service did not manage to get the credentials from the server. Those cases can be when, for example, indeni only needs ssh credentials for interrogation but the playbook requires HTTP credentials
Using psql get the last credentials used by the device:
select * from credential where id in (select credential_id from last_used_credentials where device_id ='YOUR DEVICE ID’');
If the data does not as expected - goto “Open ticket for Server team”
The error INSUFFICIENT_DATA_ACQUIRED indicated that the Auto triage service did not manage to extract enough data for the server. This error will appear when:
The playbook conclusion is empty
The number of returned tasks is 0
This behavior will happen when the playbook exited in an unexpected way without throwing an exception.
In order to debug the ansible process - goto “Reading the ansible logs”
The error PLAYBOOK_RUN_FAILED indicated that the ansible process failed while running the playbook
In order to debug the ansible process - goto “Reading the ansible logs”
The ansible logs are in /usr/share/indeni-services/logs/ansible
Enter the directory with the correct device-id
Enter the directory - artifacts
Enter the directory with the correct job-id
Look at the ‘stdout’ file in this directory. This should give the full output of the Ansible run. E.g.,
/usr/share/indeni-services/logs/ansible/<device_id>/artifacts/<job_id>/stdout |
Inside there is usually the exception of the failure.
If the failure message in the log is not clear - goto “Open ticket for Server team”
When opening a ticket please include:
Automation log file - less /usr/share/indeni-services/logs/automation.log
Server log file - less /usr/share/indeni/logs/indeni.log
Ansible artifact dir - /usr/share/indeni-services/logs/ansible
Also, make sure you write in details all the steps that you went and all the data you collected