The first version of the AWK parser uses a modified version of JAWK – an implementation of AWK in Java – and is able to support most of our functional needs, but its performance are lacking; the parser consumes too much CPU.
To improve performance, two changes have been made:

1. Instead of using the JAWK in interpreter mode, the parser now uses the compiler mode, which (a) interprets the code only once (and not on each execution) and (b) uses the JIT compiler of the JVM.
2. Instead of outputting metrics and tags via strings that are then parsed by regex, the parser now creates these objects directly from the AWK, thus removing the need to use regex parsing (which is a costly operation).

In order to use the new parser, one must run commands with the new command runner. Information on how to upgrade is available here.

New Features

Not only the performance of the parser improved dramatically, some new features were added:

• Better detection of compilation errors (although the error messages may be less clear).
• Trying to refer to a non-existent indeni variable (dynamic / double, complex, temp, etc…) is detected in compilation.

Compatibility Issues

These changes were not possible without breaking compatibility of existing commands using the AWK capabilities of the parser - either the AWK parser itself or AWK code inside _transform sections of the XML/JSON parsers.

Accessing indeni variables is no longer supported via the ${dynamic.myvar} syntax, since they were injected on runtime by replacing the code (via regex) before interpreting it, but now the code is interpreted compiled only once.

The solution was to add new AWK functions that allow access to the variables; the names of these function retain the same naming convention. So now, instead of ${dynamic.myvar}, one should use dynamic("myvar").

Note that if we were to replace all the current commands with this new syntax, leaving the rest of the code untouched, it might have caused syntax compilation errors.

For instance, the code print "hey there ${dynamic.name}" will be replaced with print "hey there dynamic("name")", which will fail due to the double quotation marks.

Also, all of these new functions are not allowed to be overridden; for instance, this code will fail to compile since it tries to override the dynamic function: dynamic = "hello".

New Functions

This is the list of the new functions available in both the AWK parser and the XML / JSON parser (in the _transform section):

• debug("message") - write a message that will printed in debug mode only (same as writeDebug).

• dynamic("variable_name") - access to a dynamic variable.

Functions available only in the XML / JSON parser:

• tag("variable_name") - access to a tag.
• double() - access to the double value of the metric.
• complex("variable_name") - access to a complex value of the metric.
• temp("variable_name") - access to a temporary variable.

Just like any other AWK keyword, one cannot define a variable / function with these names.

AWK Upgrader

A new script called awk-dsl-upgrader is now available; the script upgrades the AWK code of old commands to the new syntax.

Note that the script is a simple one and relies only on regex; thus, it cannot detect all the cases where the old syntax doesn’t work anymore.

Also, it’s probably best to make sure it doesn’t replace parts of the command that it shouldn’t.

Examples

Here are a few examples of AWK code before and after the changes:

1. Here we change the way we reference the temporary variables ps_status and psnum.

   Before:

   status = trim(tolower("${temp.ps_status}"))
   print "Power Supply ${temp.psnum}"

   
   After:

   status = trim(tolower(temp("ps_status")))
   print "Power Supply " temp("psnum")

2. Here we change the way we reference the temporary variable grp_name.

   Before:

   { print "${temp.grp_name}:${temp.grp_ip}" }

   
   After:

   { print temp("grp_name") ":" temp("grp_ip") } 

3. Here we change the name of a variable from tag to tags, since tag is now a keyword in the AWK script.
   
   Before:

   tag["name"] = name
   writeDoubleMetric("bgp-state", tag, "gauge", 300, status)

   
   After:

   tags["name"] = name 
   writeDoubleMetric("bgp-state", tags, "gauge", 300, status)