Defining a PipeLine Descriptor

A PipeLine descriptor defines the aspects of monitoring output from a process. The monitored process can be a GT.M (any version) process or any other UNIX process. The Publisher master process JOBs a PipeLine process which reads the stdout and/or stderr of the monitored process line by line, executing a GT.M expression (an extrinsic function call) for each line read in order to gather and format the per-line information. As the name indicates, a PipeLine process uses a UNIX pipe to monitor a process.

The syntax of a PipeLine Descriptor is:

PipeLine:{InfoDictID | InfoDictName}:[PipeLineName]:[PipeLineID]:{PublisherID | PublisherName}:PipeCmd:[PipeCycle]:[Timeout]:[PieceSeparator]:[PreExpr]:[InfoExpr]:[PostExpr]:

InfoDictID or InfoDictName: The InfoDict ID or Name of the InfoDict holding the PipeLine definition.
PipeLineName: The name of the PipeLine.
PipeLineID:The Publisher master process creates a GT.M PIPE device, which sets up a UNIX pipe, and uses untimed READ commands to accept stdout and stderr from the process created on the pipe.
The descriptor must specify one, or both, of the PipeLineName and PipeLineID. When the descriptor specifies both the Name and ID and there is no corresponding InfoDict entry, configuration processing implicitly creates an InfoDict entry.
If PipeLineName is specified, the configuration gets the PipeLineID from the specified InfoDict domain. If the specified domain has no definition for the Name, the InfoHub configuration process generates the PipeLineID by adding one or two to the last ID used in this InfoDict domain in order to get the next odd (for stdout) or even (for stderr) integer. Even when a PipeLine descriptor specifies gathering information only from one or the other, the InfoHub reserves the block of two numbers starting with the PipeLineID.
If PipeLineID is specified, the configuration process verifies before configuration completion that the specified InfoDict domain maps a PipeLineName; if the configuration detects there is no Name or ID for a PipeLine, it produces an IHIDORNAMEREQ error; if it detects an attempt to map multiple PipeLineIDs to one Name, it produces an IHDUPPIPELINE error.
PipeCmd: PipeCmd is shell (/bin/sh) command to launch the process which feeds results to the PIPE device. The default environment of the process includes changes specified by Env descriptors for the InfoHub as well as for the Publisher. If it is just a filename and not a path, $PATH in the environment of the Publisher must include a path to the appropriate executable filename. If it includes a path to the filename, it must be an absolute path (since PipeCmd should not count on the working directory of InfoHub). Given this, InfoHub relies on environment variables from the environment of Publisher.
Because PipeCmd can terminate while the InfoHub has it open, PipeCycle specifies what InfoHub should do if and when it happens:
TimeOut: When the PipeLine process is asked to terminate, Timeout is the maximum amount of time that PipeLine process should wait for additional information from the PipeCmd process before it shuts down. If PipeLine determines that PipeCmd process has shut down, it need not wait for additional input. If not specified, Timeout defaults to one second. If the configuration processing finds a non-numeric timeout, it produces an IHBADTIMEOUT error.
PieceSeperator: PieceSeparator is a single character (which can be a multi-byte character if the InfoHub is operating in UTF-8 mode) that specifies a separator used to separate subscripts and information returned by PreExpr, PostExpr, InfoExprOut and InfoExprErr. If PieceSeparator is not specified it defaults to the colon (":"). Since the colon separates syntactic elements of the PipeLine descriptor, the only way to specify a colon as the PieceSeparator is to default it. The PieceSeparator can also be the case-insensitive keyword "var" which specifies that the first character of the expression returned by the gleaner routine is the PieceSeparator for the Reporting process to remove and use. If the configuration processing finds the specified PieceSeparator is not any of: the empty-string, a single character or the keyword "VAR", it produces an IHBADDPSEP error.
PreExpr:A GT.M extrinsic function call, which, if specified, automatically executes before FileLine starts to process the input on its PIPE or FIFO initially. If a PipeLine processor needs to initialize some values that it should only initialize once, it must differentiate between the first and subsequent invocations. Like the other *Expr functions it can return zero or more PieceSeparator delimited values as appropriate for storage in the InfoHub database. If not specified, PreExpr defaults to the empty string.
InfoExpr: A GT.M extrinsic function call that returns zero to many (typically one) PieceSeparator delimited sets of "key<PieceSeperator>value" for each line of the monitored input read. Note that key should be the numeric ID of a defined InfoDict Item. The local variable %l is available to InfoExpr with the current line read from the PIPE or FIFO. If not specified, InfoExpr defaults to an empty string.
PostExpr: A GT.M extrinsic function call which, if specified, automatically executes at the end of processing input from the PIPE or FIFO, either when a version of PipeCmd is terminates, or at InfoHub shutdown. If a PipeLine processor needs to summarize some values that it should only summarize once, it must differentiate between the last and prior invocations. If not specified, PostExpr defaults to the empty string. Like the other *Expr functions it can return zero or more PieceSeparator delimited values as appropriate for storage in the InfoHub database.
If PreExpr, InfoExpr, or PostExpr return a value which is not a empty string or a string in the format below, the PipeLine process terminates. In order for Subscriptions to detect an error the Expr should return a "key:value" pair for the error. The normal return is an empty string (meaning no information stored), or one or more "key : value" pairs to store, in the form of one or more positive integer keys, separated by PieceSeparator, followed by the value to be stored. Multiple pairs could be (as an implementation detail) separated by pairs of PieceSeparators. Neither the key nor the value can contain a PieceSeparator.
If the Configuration File does not specify at least one of PreExpr, InfoExpr, and PostExpr, configuration processing produces an IHSOMEEXPRREQ error.

	InfoHub Internals
On startup, each process for a PipeLine descriptor executes its PreExpr, gathering any information provided and logging an error if the data does not match the expected format, as described earlier. In the event PreExpr does not return an error, the PipeLine process OPENs the PipeCmd using a PIPE device and /bin/sh. The deviceparameters for the GT.M OPEN command used to start PipeCmd includes /bin/sh syntax set the environment appropriately. PipeCmd starts with environment variable changes as specified by ^InfoHubConf(InfoHubID,a??"Publishers",a??PublisherID,a??BeginningSequenceNumber,a??"EnvSetUp") for its Publisher as well as those specified in ^InfoHubConf(InfoHubID,a??"EnvSetUp",a??BeginningSequenceNumber) for descriptors applicable to the InfoHub itself (that is, applicable to all Publishers). In the event the PipeCmd process terminates, the PipeLine process executes PostExpr, and terminates if PostExpr returns an error. Then, if PipeCycle is -1, the PipeLine process logs in the InfoHub database the fact that it is terminating and terminates. If PipeCycle is non-negative, the PipeLine process executes PreExpr, and if the return value is zero, starts a new invocation of PipeCmd. Configuration processing stores PipeLine specifications in the following node: ^InfoHubConf(InfoHubID,"Publishers",PublisherID,BeginningSequenceNumber,a??PipeLineID)=a??[EndingSequenceNumber]:a??PipeCycle:a??InfoDictID:a??PipeLineName:a??PipeCmd:a??Timeout:a??[PieceSeparator]:a??[PreExpr]:a??[InfoExpr]:a??[PostExpr]

InfoHub Internals

On startup, each process for a PipeLine descriptor executes its PreExpr, gathering any information provided and logging an error if the data does not match the expected format, as described earlier. In the event PreExpr does not return an error, the PipeLine process OPENs the PipeCmd using a PIPE device and /bin/sh. The deviceparameters for the GT.M OPEN command used to start PipeCmd includes /bin/sh syntax set the environment appropriately. PipeCmd starts with environment variable changes as specified by ^InfoHubConf(InfoHubID,a??"Publishers",a??PublisherID,a??BeginningSequenceNumber,a??"EnvSetUp") for its Publisher as well as those specified in ^InfoHubConf(InfoHubID,a??"EnvSetUp",a??BeginningSequenceNumber) for descriptors applicable to the InfoHub itself (that is, applicable to all Publishers).

In the event the PipeCmd process terminates, the PipeLine process executes PostExpr, and terminates if PostExpr returns an error. Then, if PipeCycle is -1, the PipeLine process logs in the InfoHub database the fact that it is terminating and terminates. If PipeCycle is non-negative, the PipeLine process executes PreExpr, and if the return value is zero, starts a new invocation of PipeCmd.

Configuration processing stores PipeLine specifications in the following node:

^InfoHubConf(InfoHubID,"Publishers",PublisherID,BeginningSequenceNumber,a??PipeLineID)=a??[EndingSequenceNumber]:a??PipeCycle:a??InfoDictID:a??PipeLineName:a??PipeCmd:a??Timeout:a??[PieceSeparator]:a??[PreExpr]:a??[InfoExpr]:a??[PostExpr]

Example:

PipeLine:UpTimeDict:Uptime::System:/usr/bin/uptime:15:2:$char(30):PreExpr^UptimeGleaner:InfoExpr^UptimeGleaner:PostExpr^UptimeGleaner

This example configures a PipeLine gleaner for the uptime command. This example is a part of SimpleMonitor.conf in the ULFM Reference Implementation.

PrevA	Up	A Next
A	Home	A

Defining a PipeLine DescriptorChapterA 4.A Configuring an InfoHub

Defining a PipeLine Descriptor

Defining a PipeLine Descriptor
ChapterA 4.A Configuring an InfoHub