JSON Parser Tutorial

The JSON parser is used to parse an JSON output from a REMOTE operation.

The JSON parser is slightly more tricky. It uses the YAML syntax (see here). One important thing to remember is that YAML is very sensitive to indentation. Do NOT use tab. Instead, use 4 spaces. Don't use any other number of spaces for indentation, otherwise the script won't work well.

The JSON parser supports emitting tags as well as metrics, so it can be used for both interrogation and monitoring. Here is a sample structure:

Input for this example:
{
	"total": "17",
    "max": "512"                                   
}

# The JSON parser has operators, which are used to define behavior or certain actions. 
# Operators always start with "_". Think of them as functions.

# This section must exist in monitoring scripts and must not exist in interrogation scripts
_metrics:
    -     # This is a bit tricky, each metric must be in its own section under "_metrics". Each metric has a "-" representing it.
        _tags:					  # Each metric has tags
            "im.name":				  # The "im.name" tag is the name of the metric (matches the list of reserved metrics)
                _constant: "arp-total-entries"       # "_constant" means - "take the string as is and use it"
        _value.double:            # This is the equivalent of writeDoubleMetric in AWK. It says "this metric is a double metric".
            _value: $.total        # This is the value of the metric. The "_value" operator extracts the text at the given JsonPath.
    -     # And here is another metric.
        _tags:
            "im.name":
                _constant: "arp-limit"
        _value.double:
            _value: $.max

JsonPath - is similar to XPath for XML. Specifically, we use this one: https://github.com/jayway/JsonPath . As you develop, you may want to test your JsonPaths. See https://jsonpath.herokuapp.com/ (be careful with placing confidential information there). Stay away from other sites that might not use Jayway, because the syntax sometimes differ and it's sure to give you a head ache if you mix them up.

Below are useful example scripts:

Interrogation script sample
For this input:
{
    "model": "5208XL",
    "serial": "007200003165",
    "version": "242.213.114.245"
}

We extract two elements for tags, and set two other tags specifically:
 
_tags:
    "os.name":
        _constant: "alteon-os"
    "os.version":
        _value: $.version
    "vendor":
        _constant: "radware"
    "model":
        _value: $.model
Monitoring script using the _groups operator to collect multiple DOUBLE items
For this input:
{
    "drives": [
        {"disk_id": "1", "status": "1"},    
        {"disk_id": "2", "status": "0"}
    ]
}

# We want to collect the status of the drives:
_metrics:
    -
        _groups:					   # The "_groups" operator tells the parser that there are multiple matching elements and a metric should be created for each of the matching elements. The output of this script will be two double metrics of the same type, one for each disk.
            $.drives[0:]:                # Notice this JsonPath - we are iterating over all of "drives"'s child elements 
									   # (it is an array, note the square bracket in the JSON content
                _tags:
                    name:
                        _value: disk_id
                    "im.name":
                        _constant: "hardware-element-status"
                _value.double:
                    _value: status      


 

Sometimes, you may need to manipulate the JSON content. For example, what if in the above _groups sample, the disk status was not provided as 1 or 0 in the JSON? What if it was string representing a status code?

To deal with this we have introduced the _transform operator. This allows you to write AWK code which can transform the JSON content. The _transform section supports all of the AWK helper functions, except "write*" functions. So let's expand the above _groups example and involve _transform. Note that the input XML is slightly different.

Monitoring script using the _groups operator and _transform block to collect multiple DOUBLE items (and transform content)
The following input JSON shows textual status values:
{
    "drives": [
        {"disk_id": "1", "status": "Present"},    
        {"disk_id": "2", "status": "Missing"}
    ]
}

So we need to parse the Present and Missing statuses into 1 and 0 respectively.


_vars:
    root: /response/result
_metrics:
    -
        _groups:
            "$.drives[0:]":
                _tags:
                    name:
                        _value: disk_id
                    "im.name":
                        _constant: "hardware-element-status"
                    "live-config":
                       _constant: "true"
                    "display-name":
                        _constant: "Hardware Elements State"
                    "im.dstype.displayType":
                        _constant: "state"
                    "im.identity-tags":
                        _constant: "name"
                _temp:                 # This is required for _transform operations. You basically save the JSON data in temporary variables.
                    status:            # This variable will later be referred to as "${temp.status}" in the _transform block.
                        _value: _status
        _transform:                    # Ready to transform some data! Note the indentation of this block - it is directly under the metric ("-") and parallel to "_groups". This transform block will be run for each occurrence of a group item (matched by "$.drives[0:]")
            _value.double: |           # See this weird pipe ("|")? This tells the system that an AWK block is a about to follow
                {
                    if (trim("${temp.status}") == "Present") { print "1.0" } else { print "0.0" }     # The trim() function is from the AWK helper functions
                }
 

When trying to emit complex metrics, things are generally similar. Here are a couple of examples:

Monitoring script with a simple complex metric
For this JSON:
{
	"timezone": "London/UK"                                   
}
_metrics:
    -
        _tags:
            "im.name":
                _constant: "timezone"
        _value.complex:      # Compare this with "_value.double"
            value:           # This is slightly different compared to double. In complex, we have key-value pairs.
                _value: $.timezone

The above parsing script will generate a complex value which looks like this:
{"value": "London/UK"}

Complex metric array with transform (advanced!!)
For this XML:
{
    "admins": [
        {"admin": "indeni", "from": "10.10.1.1", "type": "Web", "session-start": "11/03 01:14:51", "idle-for": "00:00:00s"},
        {"admin": "admin", "from": "10.10.1.1", "type": "CLI", "session-start": "11/03 01:13:32", "idle-for": "00:01:31s"}
    ]
}
We are trying to extract who is logged in and do some transform on the input.
_metrics:
    -
        _groups:
            $.admins[0:]:
                _tags:
                    "im.name":
                        _constant: "logged-in-users"
                _temp:
                    idlefor:              # We will need to parse the idlefor into seconds
                        _value: "idle-for"
                _value.complex:
                    "from": 
                        _value: "from"
                    "username":
                        _value: "admin"
        _transform:                       
            _value.complex:               # Note, "_value.complex" shows up twice here. The parser will merge the two together into a single complex metric entry.
                "idle": |
                    {
                        str = "${temp.idlefor}"
                        sub(/s/, "", str)
                        split(str, timearr, ":")
                        print timearr[1] * 3600 + timearr[2] * 60 + timearr[3]
                    }
        _value: complex-array             # This is a very special operator, it tells the parser to take all the items found under the "_groups" for this metric and turn it into a JSON array

So the final output looks like this:
[ {"from": "10.10.1.1", "username": "indeni", "idle": "0"}, {"from": "10.10.1.3", "username": "admin", "idle": "91"}]

If we didn't include the "_value: complex-array", the parser would have tried to create two separate metrics, one for each admin. However, the definition of the "logged-in-users" metric requires them all to be in one metric as a JSON array. 

For a full list of operators, see this.

Important Note: Missing paths

If a certain path doesn't exist, the parser will skip that metric (because it will consider the path as a failure). For example, consider this:

_metrics:
    -
        _value.double:
            _value: $.somepath.somesubelement

If "somesubelement" doesn't exist, then _value won't evaluate and the specific metric will be dropped.

But what if you still want to create the metric? The way to do that is by using _count (see the list of operators). You can "_count" the somesubelement and if you get 0 you know it's not there. For example:

_metrics:
    -
        _temp:
            subelementcount:
                _count: $.somepath.somesubelement
        _transform:
            _value.double: |
                {
                    if ("${temp.subelementcount}" == "0") { print "0" } else { print "1" }
                }

Troubleshooting

"No metrics nor dynamic variables were parsed from the tree"

This message indicates that no metrics was found.

Check your:

  • Paths to values
  • Paths to variables
  • Paths to tags
  • Make sure that the rest URI contains the data you want 
  • That any existing AWK code is correct

"AWK code isn't a string but: YamlObject(Map(YamlString(_value) -> YamlString"

You have most likely used the _value: command under a _transform section, but _value is used for fetching data from directly from json structure whereas _transform is expecting data from awk.

Examples:

#Correct:
_transform:
	_value.double: |
		{
				print "1"
		}
	_tags:
		"name": |
			{
				print "mytag"
				}


#Wrong:
_transform:
	_value.double:
		_value: |
			{
				print "1"
			}
	_tags:
		"name":
			_value: |
				{
					print "mytag"
				}




Good practices

It is recommended to put the sections in this order if possible:

  • _tags
  • _temp (if available)
  • _transform (if available)
  • _value (last)

Reason being:

  • _tags first because it gives the reader a clue about what he/she is about to read.
  • _temp is used in the transform section and thus should precede it for a natural way of reading as the values here is used later on
  • _transform should come after _temp because of the previous bullet
  • _value comes last, because all of the logic above has been used to provide this part.

Within _tags "im.name" should be the first one.