Using python as a parser, the IKE will have the power to:
Use any tutorial that matches your OS, for example https://realpython.com/installing-python/
make sure the installation is a success by running on the terminal
python3 --version |
The result should be
Python 3.7.4 |
Using the terminal, install the latest version of indeni-parser
pip3 install "indeni_parser==0.0.0.*" --extra-index-url https://indeni.jfrog.io/indeni/api/pypi/indeni-pypi-develop/simple |
The result should be similar to:
Looking in indexes: https://service.indeni-ops.com/repo/api/pypi/indeni-pypi/simple Collecting indeni-parser Downloading https://service.indeni-ops.com/repo/api/pypi/indeni-pypi/indeni-parser/0.0.0.1667/indeni_parser-0.0.0.1667-py3-none-any.whl Installing collected packages: indeni-parser Successfully installed indeni-parser-0.0.0.1667 |
pip3 uninstall indeni-parser |
Install the community edition of PyCharm
https://www.jetbrains.com/pycharm/download
The Project Interpreter should be:
/usr/local/bin/python3.7 |
Continue for next step in a case that the Project Interpreter is not as mentioned or it's one of:
/usr/local/bin/python2 - using python2 /usr/local/bin/python - using default python /document/venv/bin/python - using virtual env |
Link to the python folder For example
/usr/local/bin/python3.7/ |
Copy the input file to a new text file and place it under the project, for example:
input_0_0:
{ "entries": [ { "id": "cpu-1", "total": "2000", "used": "300" }, { "id": "cpu-2", "total": "2000", "used": "1000" }, { "id": "cpu-3", "total": "2000", "used": "1500" }, { "id": "cpu-4", "total": "2000", "used": "10" } ] } |
On menu go to File → New.. → IndeniParser
Give names to the file and class.
The file name cannot contain dots (.) use -/_ instead (e.g. mpstat_parser.py)
The class name should be UpperCamelCase for example MpstatParser.
The first thing we need to do is to take the raw data and convert it to python object.
There are several ways to do the conversion:
Warning: when parsing an XML list with one value, we don't have a way to know this is actually a list, and therefore the parser returns the result as value and not a list.
In order to avoid exceptions, you should always wrap list values from XML with the method to_list, for example,
helper_methods.to_list(result['entries']) |
As an example, we will simply use the json parser:
data = helper_methods.parse_data_as_json(raw_data) |
After getting the data, we can process it to get the values that we need.
This section will normally be just a pure python, with the use of common methods that under the library helper_methods.
In our example we would like to take the data and create a map between cpu-id and percentage used:
for entry in entries: cpu_id = entry["id"] cpu_used = float(entry["used"]) cpu_total = float(entry["total"]) cpu_used_in_percentage = cpu_used / cpu_total * 100 cpus[cpu_id] = cpu_used_in_percentage |
Finally, we need to report the data.
We have several types of data to report:
By hovering the method or click it you will get info what data to enter.
In our example, we want to report double metric for each cpu
for cpu_id in cpus: tags = {"name" : cpu_id} self.write_double_metric("cpu-usage",tags, "gauge", cpus[cpu_id], False) |
The parser file should look like this:
from parser_service.public import helper_methods from parser_service.public.base_parser import BaseParser class CpuParser(BaseParser): def parse(self, raw_data: str, dynamic_var: dict, device_tags: dict):): self.debug("start parser") # Step 1 : Data Extraction data = helper_methods.parse_data_as_json(raw_data) # Step 2 : Data Processing entries = data['entries'] cpus = {} for entry in entries: cpu_id = entry["id"] cpu_used = float(entry["used"]) cpu_total = float(entry["total"]) cpu_used_in_percentage = cpu_used / cpu_total * 100 cpus[cpu_id] = cpu_used_in_percentage # Step 3 : Data Reporting for cpu_id in cpus: tags = {"name" : cpu_id} self.write_double_metric("cpu-usage",tags, "gauge", cpus[cpu_id], False) return self.output # Test your code Here # helper_methods.print_list(CpuParser().parse_file("input.json",{})) |
(You can see that we added Debug message which can help the IKE find issues)
Uncomment the last line and run the class, fix the input name to the name you created and run the class, the result should be:
{'action_type': 'Debug', 'timestamp': 1566803012214, 'message': 'start parser'} {'action_type': 'WriteDoubleMetric', 'timestamp': 1566803012214, 'tags': {'name': 'cpu-1'}, 'value': 15.0, 'name': 'cpu-usage', 'ds_type': 'gauge'} {'action_type': 'WriteDoubleMetric', 'timestamp': 1566803012214, 'tags': {'name': 'cpu-2'}, 'value': 50.0, 'name': 'cpu-usage', 'ds_type': 'gauge'} {'action_type': 'WriteDoubleMetric', 'timestamp': 1566803012214, 'tags': {'name': 'cpu-3'}, 'value': 75.0, 'name': 'cpu-usage', 'ds_type': 'gauge'} {'action_type': 'WriteDoubleMetric', 'timestamp': 1566803012214, 'tags': {'name': 'cpu-4'}, 'value': 0.5, 'name': 'cpu-usage', 'ds_type': 'gauge'} |
command-runner working with Python in is the same way as with AWK.
See Command Runner.
The only thing needed to do, in order to use the python parser file over indeni platform, is to link the ind script to the parser and choose the right type:
- run: type: SNMP command: GETBULK --columns 1.3.6.1.4.1.9.9.48.1.1.1.2 1.3.6.1.4.1.9.9.48.1.1.1.5 1.3.6.1.4.1.9.9.48.1.1.1.6 parse: type: PYTHON file: memory_usage_parser.py |
As long as you installed the indeni-parser package the command-runner should work
The test files for each parser should be located in a test folder, next to the parser file.
For example:
indeni-knowledge/parsers/src/panw/panos/panos_anti_spyware_info_low_severity/test |
The test file should start with the prefix test_ for example:
indeni-knowledge/parsers/src/panw/panos/panos_anti_spyware_info_low_severity/test/test_panos_anti_spyware_info_low_severity_parser_1.py |
In order for python to recognize the path of the packages, the path should not contain any folder/file name with "-", it can only contain "_".
while running the test, pycharm needs to know that src folder is a source folder. we can do it by right-clicking:
indeni-knowledge/parsers/src/ |
and choose Mark directory as Sources root
The folder color should be turned to blue.
example for test file:
import os import unittest from panw.panos.panos_anti_spyware_info_low_severity.panos_anti_spyware_info_low_severity_parser_1 import LowSevParser1 from parser_service.public.action import WriteDynamicVariable class TestLowSevParser1(unittest.TestCase): def setUp(self): # Arrange self.parser = LowSevParser1() self.current_dir = os.path.dirname(os.path.realpath(__file__)) def test_valid_input(self): # Act result = self.parser.parse_file(self.current_dir + '/valid_input.xml', {}, {}) # Assert self.assertEqual(2, len(result)) self.assertTrue(isinstance(result[0], WriteDynamicVariable)) self.assertEqual('profile', result[0].key) self.assertEqual('Test', result[0].value) self.assertTrue(isinstance(result[1], WriteDynamicVariable)) self.assertEqual('profile', result[1].key) self.assertEqual('Test-1', result[1].value) def test_invalid_input(self): # Act result = self.parser.parse_file(self.current_dir + '/invalid_input.xml', {}, {}) # Assert self.assertEqual(0, len(result)) if __name__ == '__main__': unittest.main() |
self.current_dir = os.path.dirname(os.path.realpath(__file__)) |