FIS InfoHub

Your InfoHub distribution has three parts – the InfoHub Product, Reference Implementation examples developed using the InfoHub Product, and an SNMP Reporting Adaptor. To set up InfoHub to monitor GT.M, read the following sections in the specified order– InfoHub Overview, InfoHub Concepts, InfoHub Tasks, Appendix A: FIS GT.M SNMP Plugin, and Appendix B: Reference Implementations. These sections help you understand the basics of InfoHub and customize Reference Implementation examples.

To use InfoHub to build a general-purpose monitoring and alerting application, read the following sections in the specified order–InfoHub Overview, InfoHub Concepts, InfoHub Tasks, Appendix A: FIS GT.M SNMP Plugin, Appendix B: Reference Implementations, and Configuring an InfoHub. Also, go through the M code of all the reference implementation example gleaners and review their Configuration files to understand how they manage monitoring and alerting.

Notes marked as InfoHub Internals are meant for those who plan to support InfoHub or develop a fairly advanced understanding of InfoHub. Other readers can skip the InfoHub Internals notes.

If you are in a hurry and just want to see a quick demonstration of how InfoHub works, proceed directly to the Uptime and Log File Monitoring Reference Implementation section and follow the step-by-step installation instructions provided in that section.

The FIS Information Hub (InfoHub) is a general-purpose application to gather, store, access, and monitor information. It is a "Hub" because it provides a central location for storing current and historical information from multiple environments and from multiple sources in each environment. InfoHub is "general-purpose" because it can be configured for use by tools via SNMP or any connector API for analytics, trending, and reporting. InfoHub is an application written in GT.M, which drives its monitoring framework and provides an n-tuple key-value datastore to hold information. InfoHub includes a ready-to-run Reference Implementation for monitoring GT.M instances that run V6.0-002 or above.

An InfoHub installation contains one or more of:

InfoHubs are general-purpose — one InfoHub can monitor multiple data sources, and a single data source can be monitored by multiple InfoHubs. The operations of an InfoHub are data-driven and defined in a text file called InfoHub Configuration File. The InfoHub Configuration File contains a series of descriptors. Descriptors define Publishers that provide information to gather and store in the InfoHub, and Subscriptions for Subscribers to alert based on the information stored in InfoHub. Descriptors also contain information about the environment, sources of data, and the frequency of monitoring (periodic, continuous, or on-demand).

Because it uses character functions for processing external data and only uses byte functions in cases where its internal operation makes them appropriate, an InfoHub can monitor data from sources that rely on ASCII or on various UTF encoding modes. As an application, it can run either in M mode or UTF-8 mode. It does restrict names in its configuration to ASCII, but it can monitor, process and store UTF-8 or UTF16 data.

InfoHub is a GT.M application. Like any other GT.M application, its database can be journaled, recovered, and replicated. An appropriately configured InfoHub using a reference implementation can monitor its own database. InfoHub includes ready-to-run reference implementations: GT.M Monitoring Reference Implementation and Uptime and Log File Monitoring (ULFM) Reference Implementation. For more information on the Reference Implementations available with the InfoHub Distribution, refer to Appendix B: Reference Implementations.

The following illustration is a Reference Implementation of InfoHub for monitoring operator log and WebServer log, and the process output from df and free shell commands. This illustration is only an example to explain a possible InfoHub framework in action. The InfoHub framework setup varies depending on the monitored components and requirements.

In this illustration, the white lines represent process controlled connections used for monitoring and the black lines represent the flow of data. The Instance A Publisher uses gleaners to gather data from the system log, WebServer log, and process output from df and free, to store in the InfoHub database.

The relationship between an InfoHub and an Adaptor is of two types–Subscriber and Query. The red line represents a Subscriber relationship where the Reporting Adaptor registers itself as a Subscriber with the InfoHub. When data arriving at the InfoHub matches a specified Subscription condition, InfoHub places the data in a queue and sends an interrupt (via a configured GT.M trigger) to the Subscriber. The Subscriber then fetches the data from a queue in the InfoHub database where InfoHub placed it before sending the interrupt. If the Subscriber is the last interested Subscriber, it removes the data from the queue.

The black line represents a Query relationship where the Reporting Adaptor retrieves the requested data from the InfoHub repository . The data flows from the InfoHub to the Subscriber, and the entire flow is managed by the Reporting Adaptor - InfoHub is not aware that the Reporting Adaptor exists.

A single Adaptor can have either one or both relationships with InfoHub. The following illustration demonstrates the relationships between InfoHub and Adaptor(s).

Note

InfoHub runs on GT.M V6.0-002 or above. To monitor log files, InfoHub requires the the FOLLOW deviceparameter which was first introduced in V6.0-002. The FOLLOW deviceparameter enables monitoring of sequential disk files that are concurrently being updated by other processes. The GT.M Monitoring Reference Implementation monitors GT.M V6.0-002 using the $ZPEEK() function, which extracts state information from database/journal files. However, using GT.M programming and shell scripting, it is possible to modify that reference implementation to monitor any GT.M version prior to V6.0-002. Required modifications deal with the format and content of file header listings, messages, and gleaning techniques that vary from release to release. For example, to monitor an earlier release which lacks the $ZPEEK() function, you have to use other techniques, such as parsing DSE output, for example, with a program such as the %DSEWRAP utility. However, you do not need to create an alternative functionality for the FOLLOW deviceparameter because it is available within InfoHub as part of the V6.0-002 and can be used for sequential disk files created by any prior GT.M version.

InfoHub is not limited to monitoring GT.M applications. Its ability to gather information and interface with other Reporting Adaptors makes it a useful tool for a broad range of monitoring and alerting needs including (but not limited to):

This section explains InfoHub concepts and roles of all components of InfoHub.

This section contains information about installing and creating an environment configuration for InfoHub.

$ cd /opt

$ tar zxvf /Distrib/InfoHub/InfoHub.63344_62100.tar.gz

$ mkdir /opt/InfoHub/configs

You can interact with InfoHub from the shell and from within GT.M. To interact with an InfoHub from the shell, the general command is:

$gtm_dist/mumps -run InfoHub --action=<Action> [--<Action_parameters>[=values]...] [--infohub=InfoHubID|InfoHubName] [--gbldir=/path/to/globaldirectory]

To interact with an InfoHub from within GT.M, invoke an entryref like:

<Action>^InfoHub(InfoHubID[,values])

InfoHub Internals

^InfoHubConf("GlobalsDirs",$zgbldir) contains the ID of the most recently configured InfoHub. With a few exceptions, e.g the Actions is "status", "list", or "configure", the underlying mechanism for taking action on an existing InfoHub is to place information in the ^InfoHubActivity global and signal the InfoHubMain service process to invoke its $ZINTERRUPT, which in turn causes it to look in ^InfoHubActivity for instructions on which to act. The action initiating process returns to the operator once it sees an acknowledgement that the InfoHubMain service has taken over the action. Although FIS plans to maintain stability of the published CLI in future releases of InfoHub, without prior notice, FIS may change the underlying mechanism within any InfoHub release. When starting an InfoHub from the shell command line or from GT.M using the appropriate entryref, the InfoHubMain launches with a JOB command, and control returns to the invoking process once the InfoHub has successfully launched.

	InfoHub API
	More details about the InfoHub API are also documented in the file InfoHub/InfoHub.m. Comments before each entryref describe the input parameters and their types.

Related Information:

To load a configuration file on an InfoHub from the shell, execute a command like:

$ $gtm_dist/mumps -run InfoHub --action=configure --file=<ConfigFileName> [--verbose[=filename]]

Attempting to configure a file that can't be opened for reading, or is defective, produces an IHBADFILE error. Attempting to configure an InfoHub with an improperly installed gtmposix plug-in may produce an IHGETTIMEFAIL error.

To load a configuration file on InfoHub from within GT.M, invoke the following entry point in the InfoHub API as an extrinsic function:

$$configure^InfoHub("InfoHubID","/path/to/ConfigFileName")

If the configuration file contains an InfoHub descriptor, the InfoHubID is optional and, if supplied, must match the one in the descriptor.

	InfoHub Internals
	As IDs, and possibly Names, may have meaning for tools that report on ^InfoHubInfo data, the Infohub configuration processing rejects attempts to change the association between an ID and a Name. A user with sufficient understanding of, and access to, the InfoHub database can change such associations programmatically using GT.M code.

Example:

$ $gtm_dist/mumps -run InfoHub --action=configure --file=samples/SimpleMonitor.conf

This example uses the CLI for InfoHub to load an InfoHub Configuration File.

GTM>if $$configure^InfoHub("","samples/SimpleMonitor.conf")

This example loads the server1.conf Configuration File on InfoHub.

GTM>if $$configure^InfoHub("","samples/SimpleMonitor.conf") 
Loading the configuration file samples/SimpleMonitor.conf
GTM>if $$configure^InfoHub("","samples/SimpleMonitor.conf") 
Loading the configuration file samples/SimpleMonitor.conf 
Configuration unchanged 
GTM>

This example attempts to load the same SimpleMonitor.conf twice using 7421 as the InfoHub ID.

Related Information:

To list all InfoHubs available in the current global directory, execute a command like:

$ $gtm_dist/mumps -run InfoHub --action=list 

To list all InfoHubs available in the current global directory from within GT.M, invoke the following entry point in the InfoHub API as an extrinsic function:

$$list^InfoHub()

Example:

$ $gtm_dist/mumps -run InfoHub --action=list 

GTM>if $$list^InfoHub()

Both examples list all InfoHubs available in the current global directory.

Related Information:

To extract the contents of an InfoHub Configuration File loaded for an InfoHub, execute a command like:

$ $gtm_dist/mumps -run InfoHub --action=extract [--configuration=[0|1|2|3|..] 

This command writes the configuration to standard output. The contents of "Included" files are also in the output.

--configuration specifies the Configuration Number of the InfoHub Configuration File. InfoHub associates a unique Configuration Number with every correctly configured InfoHub Configuration File loaded for an InfoHub. When configuration is not specified or is specified as 0, InfoHub produces a historic listing of all correctly configured InfoHub Configuration Files that were loaded for an Infohub. The listing includes Configuration Numbers, location of the InfoHub Configuration Files, and the time when they were loaded on InfoHub.

To extract an InfoHub configuration file from within GT.M, invoke the following entry point in the InfoHub API as an extrinsic function:

$$extract^InfoHub(InfoHubID,0|1|2|3|..)

Example:

$ $gtm_dist/mumps -run InfoHub --action=extract
Configuration 1 was installed on 1383129299046237 from the file samples/SimpleMonitor.conf
Configuration 2 was installed on 1383132419190354 from the file samples/SimpleMonitor.conf
Configuration 3 was installed on 1383132426677351 from the file samples/SimpleMonitor.conf

GTM>if $$extract^InfoHub("11",0)

Both examples produce a historic listing of all correctly configured InfoHub Configuration Files that were loaded for an InfoHub.

Related Information:

To start an InfoHub from the shell, execute a command like:

$ $gtm_dist/mumps -run InfoHub --action=start 

To start an InfoHub from within GT.M, invoke the following entry point in the InfoHub API as an extrinsic function:

$$start^InfoHub(InfoHubID,$zgbldir)

Example:

$ $gtm_dist/mumps -run InfoHub --action=start

This example starts the InfoHub most recently associated with the current global directory.

GTM>if $$start^InfoHub(7421)
JOB 10215 InfoHub Active at 1383133001659001

This example starts the InfoHub whose ID is 7421.

Related Information:

To check the status of an InfoHub from the shell, execute a command like:

$ $gtm_dist/mumps -run InfoHub --action=status [--full] 

To check the status of an InfoHub from within GT.M, invoke the following entry point in the InfoHub API as an extrinsic function:

$$status^InfoHub(InfoHubID,{0|1})

0 specifies one line summary of the current status and health of the specified InfoHub and 1 is equivalent to the --full option.

Example:

$ $gtm_dist/mumps -run InfoHub --action=status 

This example displays the one line summary on the status and health of the most recent InfoHub.

GTM>if $$status^InfoHub(7421,1)
InfoHub ID      |InfoHub Name   |State
7421            |DEMO           |Running
Process listing
   10215 InfoHub       /usr/lib/fis-gtm/V6.0-002/mumps -direct InfoHub ^InfoHubMain 7421 /opt/InfoHub/InfoHub.gld
   10222   NoInfo      /usr/lib/fis-gtm/V6.0-002/mumps -direct InfoHub ^InfoHubNoInfo 7421 04261677 3 /opt/InfoHub/InfoHub.gld
   10217   Publishers  /usr/lib/fis-gtm/V6.0-002/mumps -direct InfoHub ^InfoHubPublisher 7421 321 /opt/InfoHub/InfoHub.gld
   10305    FileLine   /usr/lib/fis-gtm/V6.0-002/mumps -direct InfoHub ^InfoHubFileLine 7421 321 3 123 /opt/InfoHub/InfoHub.gld

This example displays the status of InfoHub ID 7421 and produces a process listing of InfoHub processes.

Related Information:

To pick up a configuration revision and restart an InfoHub, execute a command like:

$ $gtm_dist/mumps -run InfoHub --action=restart 

To pick up a configuration revision and restart an InfoHub from within GT.M, invoke the following entryref:

restart^InfoHub(InfoHubID|InfoHubName)

This command stops all components whose configuration has changed and starts all components whose configuration has changed (which might not be the same set); attempting to restart of a non-running InfoHub produces an IHNOTRUNNING error.

	Important
	The restart command also clears prior restart history for each monitored process. Service monitors will not restart a configuration that fails to start five times within five minutes.

Related Information:

To effect a clean InfoHub shutdown from the shell, execute a command like:

$ $gtm_dist/mumps -run InfoHub --action=shutdown 

To effect a clean InfoHub shutdown from within GT.M, invoke the following entry point in the InfoHub API as an extrinsic function:

$$shutdown^InfoHub(InfoHubID|InfoHubName|^InfoHubConf("GlobalsDirs",$zgbldir))

	Important
	Although an InfoHub can be stopped with the MUPIP STOP command, FIS recommends against using MUPIP STOP to shut down an InfoHub because it leaves orphaned service processes.

Example:

$ $gtm_dist/mumps -run InfoHub --action=shutdown

This example shuts down the InfoHub most recently updated using the current global directory.

GTM>if $$shutdown^InfoHub(7421)

This example shuts down the InfoHub having 7421 as the InfoHub ID.

InfoHub Internals

On receipt of a shutdown command (for example, from mumps -run InfoHub --action=shutdown), the InfoHubMain process sends a shutdown message (using the interrupt-based mechanism) to each NoInfo and Publisher process. Each Publisher process in turn sends similar shutdown messages to monitored FileLine and PipeLine processs. FileLine processs acknowledge receipt, switch their file access mode from FOLLOW to NOFOLLOW with a USE command, continue processing until the end-of-file or Timeout elapses, log their shutdown, and then halt. For a PipeLine process, the Publisher process sends it "shutdown" message. The PipeLine process sends a "shutdown message" to the PipeCmd process. The PipeCmd process does a READ x:1 at the start of every pass through for data collection. If it reads something as opposed to the 1 second timeout, the sub-process terminates immediately. If the process created by the PipeCmd is a GT.M process, it must READ or WRITE to its principal device in order to detect that PipeLine process has closed the pipe; it should then HALT. Each Publisher process waits to receive acknowledgement and shutdown completion status from each monitored xLine processes. Only then does it logs its own shutdown, and shuts down. When the InfoHubMain receives acknowledgements from all NoInfo and Publisher processes, it logs its shutdown, and shuts down. The shell command or extrinsic function call then completes.

Related Information:

To perform an InfoHub rundown from the shell, execute a command like:

$ $gtm_dist/mumps -run InfoHub --action=rundown 

After a shutdown caused by something other than an InfoHub "shutdown" action, such as a MUPIP STOP, or a system crash, or a switch-over to a new primary, a rundown cleans up the database.

To perform an InfoHub rundown from within GT.M, invoke the following entry point in the InfoHub API as an extrinsic function:

$$rundown^InfoHub(InfoHubID)

Related Information:

To debug an InfoHub from the shell, execute a command like:

$gtm_dist/mumps -run InfoHub --action=jobexam 
 [--pidlist=<comma,delimited,list,of,PIDs>]

To debug an InfoHub from within GT.M, invoke the following entry point in the InfoHub API as an extrinsic function:

$$jobexam^InfoHub(InfoHubID,"comma,delimited,list,of,PIDs")

Example:

GTM>if $$status^InfoHub(7421,1)
InfoHub ID      |InfoHub Name   |State
7421            |DEMO           |Running
Process listing
   28981 InfoHub       /usr/lib/fis-gtm/V6.0-003_x86_64/mumps -direct InfoHub ^InfoHubMain 7421
   28987   NoInfo      /usr/lib/fis-gtm/V6.0-003_x86_64/mumps -direct InfoHub ^InfoHubNoInfo 7421 1998603323 1 /home/jdoe/infohub/InfoHub/InfoHub.gld
   28985   Publishers  /usr/lib/fis-gtm/V6.0-003_x86_64/mumps -direct InfoHub ^InfoHubPublisher 7421 321 1 /home/jdoe/infohub/InfoHub/InfoHub.gld
   29085    FileLine   /usr/lib/fis-gtm/V6.0-003_x86_64/mumps -direct InfoHub ^InfoHubFileLine 7421 321 1 123 /home/jdoe/infohub/InfoHub/InfoHub.gld
GTM>if $$jobexam^InfoHub(7421,"28981,28987")
PID [28981] has been notified
PID [28987] has been notified
GTM>

$ $gtm_dist/mumps -r ^InfoHub --action=jobexam --pidlist=28981,28987
PID [28981] has been notified
PID [28987] has been notified
$

Both examples place the output of ZSHOW "*" in a file called infohub-<service>-<infohub id>[-<publisher | noinfo id>[-<xline id>]-<configuration sequence number>.ZSHOW_DMP_"_$JOB_"_"_<cntr> for PIDs 28981 and 28987.

	Important
	The ZSHOW "*" output in the JOBEXAMINE files may contain confidential information gleaned from the monitored components that the process holds in local variables, and, possibly, in Intrinsic Special Variables (ISVs). If that is the case in your InfoHub deployment, ensure that the files produced by this command are appropriately secured.

Related Information:

To purge InfoHub entries from the shell, execute a command like:

$ $gtm_dist/mumps -run InfoHub --action=purge --before=<time> [--activity] [--config] [--info] [--all]

To purge InfoHub entries from within GT.M, invoke the following entry point in the InfoHub API as an extrinsic function where 'TIME' is either a Unix time integer and 'options' is a MUMPS variable with subscripts "InfoHubActivity", "InfoHubConf", "InfoHubInfo" set to a positive integer value indicating which globals to purge:

$$purge^InfoHub(InfoHubID,TIME,options)

Example:

$ $gtm_dist/mumps -run ^InfoHub --action=purge --config --before=\"30-OCT-2013 09:00:00\"

This example removes all InfoHub configuration entries that were added before October 30, 2013 at 9 am UTC.

GTM>set (opt("InfoHubActivity"),opt("InfoHubConf"),opt("InfoHubInfo"))=1 if $$purge^InfoHub(7421,1383138000,opt)

This example removes all entries related to configuration, InfoHub Activity, and InfoHub data that was added before 30-OCT-2013 09:00:00 UTC.

Related Information:

Syntax:

InfoHub:[InfoHubName]:[InfoHubID]

InfoHub Internals

InfoHubID applies to all elements in a configuration file and is the first subscript for all global variables in the database related to a particular InfoHub, except for the sub-tree that maps global directories to InfoHubIDs using "GlobalsDirs" as the first subscript. Configuration processing stores InfoHub descriptor information in the following node: ^InfoHubConf(InfoHubID,BeginningSequenceNumber)=​[EndingTimeStamp]:​BeginingTimeStamp:​InfoHubName:​GldFileName:​ConfigurationFileName

InfoHub Internals

Processing a Configuration File is done using a GT.M LOCK on ^InfoHubConf with InfoHubID as a subscript to ensure only one process is handling a Configuration File for an InfoHub at a given time. If the configuration process detects an another process is holding a lock on this InfoHubID for more than 30 seconds, it produces an IHCONFLOCKED error. The Verbose option for configuration processing generates substantial output that documents how the processing works. This may be useful for debugging some types of configuration problems. If you have a rich configuration, consider building it up in a series of modest increments to make debugging easier.

Example:

InfoHub:SimpleMonitor:11

This example defines an InfoHub called SimpleMonitor. This example is a part of SimpleMonitor.conf configuration file in the ULFM Reference Implementation.

Related Information:

Include files are fragments of a configuration file that permit modularization of configuration information. Configuration processing parses them as it reads their Include Descriptors. If configuration processing encounters a problem reading an Include file, the configuration fails with an IHBADFILE error. The format of an Include Descriptor is:

Inc:Include:file-path specification

The file path specification can be either an absolute path or a relative path. Relative paths are relative to the configuration file and not the current working directory.

On a system that has many independent things to monitor, use Include files to make it easy to activate and deactivate them in a configuration,

Example:

Include:InfoDictConf0.conf

Related Information:

Zero or more Env descriptors can immediately follow the InfoHub descriptor, or start the configuration definition if the InfoHub descriptor is implicit; and zero or more Env descriptors can immediately follow any Publisher descriptor. Those in the first location define common environment variables for all processes used by the InfoHub; those in the second location define common environment variables for all PipeLine generated processes subject to that Publisher. The syntax of an Env Descriptor is:

Env:EnvVarName[=[Value]][,,,,] 

If the configuration processing encounters an Env descriptor that does not follow the InfoHub descriptor or a Publisher descriptor, it produces an IHENVPLACE error. There are three types of Env descriptor—one sets values in the environment of InfoHub processes, another with an equal-sign but no value ensures environment variables are not set (that is, cleared), and a third to invoke a shell script that manages the target environment. For example:

Attempting to configure an env descriptor without providing a suitable environment produces an IHBADENV error. Each EnvVarName is a legal POSIX environment variable name, consisting solely of ASCII letters, digits, and underscores ("_"), of which the first character must not be a digit; if configuration processing encounters a failure to meet this criterion, it also produces the IHBADENV error. To allow for permissible, non-graphic values^[1] of environment variables, and to allow environment variables to reference environment variables of the InfoHub itself (using the $ZTRNLNM() function), each Value is treated as a GT.M expr that the InfoHub evaluates before storing the result in its database.

InfoHub Internals

Configuration processing stores Env descriptor information in the following nodes: ^InfoHubConf(InfoHubID,"EnvSetUp",1,BeginningSequenceNumber)=​[EndingSquenceNumber]:​env=[value],..., where the string value of the node is constructed of shell commands that an InfoHub process uses to manage its environment. ^InfoHubConf(InfoHubID,"Publishers",PublisherID,BeginningSequenceNumber,​"EnvSetUp")​=[EndingSquenceNumber]:​env=[value],..., where the string value of the node is constructed of shell commands that a Publisher process uses to adjust the environment provided to it by its InfoHub to be appropriate for a PipeLine process. Expression evaluation (implemented with indirection or XECUTE) also has the potential for side effects that can affect the InfoHub itself. To protect the InfoHub itself from side effects of evaluated expressions, the InfoHub processes Env descriptors in a routine which protects itself by: setting $ZROUTINES to the null string to prevent any new routines from being linked; creating a temporary global directory and database with $ZGBLDIR pointing there; executing an exclusive NEW; and receiving results in $ZTWORMHOLE, it being impossible to modify $ZTWORMHOLE as a side-effect of an expression. ($ZTWORMHOLE can itself be safely saved and restored in the calling routine to retain its originally intended use for triggers.)

Example:

Env:LC_TIME='fr_FR.UTF-8'
Env:LANG='en_US.utf-8'

Related Information:

The InfoDict [Domain] Descriptor can appear anyplace after the InfoHub Descriptor, except immediately prior to an ENV descriptor, but most logically falls between any InfoHub Environment Descriptors and the Publisher Descriptors. An InfoDict is an organizational container or domain. While InfoDicts are the tool for organizing the schema and behavior of the InfoHub, they do so indirectly, and never appear in the paths of the main storage schema. The syntax of an InfoDict Domain Descriptor is:

InfoDict:InfoDictName:[InfoDictID][:{ParentInfoDictID | ParentInfoDictName}]

InfoHub Internals

Configuration processing stores InfoDict domain information in the following nodes: ^InfoHubConf(InfoHubID,"InfoDicts",InfoDictID,BeginningSequenceNumber)=​EndingSequenceNumber]:​InfoDictName ^InfoHubConf(InfoHubID,"InfoDicts",InfoDictID,BeginningSequenceNumber,0,​ParentInfoDictID)​=[EndingSequenceNumber]: ^InfoHubConf(InfoHubID,"InfoDicts",InfoDictID,BeginningSequenceNumber,​"Children",​ChildInfoDictID): this node is only maintained for the current configuration of this InfoHub.

Example:

InfoDict:GenericDict::SystemDict
InfoDictItem:GenericDict:OpLog:1137
InfoDictItem:GenericDict:AuthLog:1139

This example defines an InfoDict Domain called GenericDict and places two InfoDict Items—OpLog and AuthLog—under it. This example is a part of SimpleMonitor.conf configuration file in the ULFM Reference Implementation.

Related Information:

The InfoDictItem Descriptor can appear anywhere after the InfoHub Descriptor, except before an Env Descriptor, but most logically falls among or after any InfoDict [domain] Descriptors and before the Publisher Descriptors. InfoDictItems form the set of nodes the InfoHub can store. While you can create them explicitly, the configuration process can implicitly create InfoDictItems corresponding to some other InfoHub components (Publishers, xLines, Subscribers). The syntax of an InfoDictItem Descriptor is:

InfoDictItem:{InfoDictID | InfoDictName}:ItemName:[InfoDictItemID]:[Label]:[Type][;ItemDescription]

	Note
	When the same InfoHub has multiple adapters, the user-defined Types must be a superset of the Types required by all the adapters, and the transforms for each adapter must deal appropriately with all Types, including transforming those outside their set into those within their set.

InfoHub Internals

Configuration processing stores InfoDict item information in the following nodes: ^InfoHubConf(InfoHubID,"InfoDicts",InfoDictID,BeginningSequenceNumber,ItemID)=[EndingSequenceNumber]:ItemName;Label:[Type]:[ItemDescription] ^InfoHubConf(InfoHubID,"InfoDicts","Dnames",InfoDictID,ItemName)=ItemID ^InfoHubConf(InfoHubID,"InfoDicts","Dnames",InfoDictName)=InfoDictID ^InfoHubConf(InfoHubID,BeginningSequenceNumber,"Paths",int1...)=ItemName;Label:[Type][:ItemDescription]

Example:

InfoDict:UpTime::UpTimeDict
InfoDictItem:UpTime:Days:3030400::Integer:Days since last reboot
InfoDictItem:UpTime:Load01:3030401::Float:One minute load average
InfoDictItem:UpTime:Load05:3030405::Float:Five minute load average
InfoDictItem:UpTime:Load15:3030415::Float:Fifteen minute load average

This example defines an InfoDict Domain called UpTime and 4 InfoDict Items called Days, Load01, Load05, and Load15. This example is a part of SimpleMonitor.conf configuration file in the ULFM Reference Implementation.

Related Information:

Information stored in an InfoHub is associated with information Publishers, each of which is specified with a Publisher descriptor. Publisher descriptors follow the InfoHub descriptor and any of its associated Environment Descriptors. While the meaning of "Publisher" is a function of a configuration and its conventions in your organization, it is appropriate for one Publisher to manage the information gathering and dissemination for one environment. This provides the PreExpr, PostExpr, and InfoExpr expressions of FileLine and PipeLine processes with an isolated environment where, if appropriate, they can communicate and cooperate with each other.

InfoHub JOBs a process for each Publisher. This process in turn JOBs processes for each FileLine or PipeLine descriptor associated with the Publisher. The Publisher process may optionally create a temporary directory before launching any processes. Within that temporary directory, it creates a global directory that maps to an unjournaled database in that temporary directory. It logs the name of the temporary directory to the InfoHub database.

The Publisher JOBs the FileLine or PipeLine processes with the following JOB Processparameters:

The syntax of a Publisher Descriptor is:

Publisher:{InfoDictID|InfoDictName}:[PublisherName]:[PublisherID]:[APIDir]:[TempPWD]:[TempDBAlloc]:[TempDBExtend]

InfoHub Internals

The entryref used to start the FileLine and PipeLine processes includes a parameter for the Publisher process to pass its own global directory, which the FileLine and PipeLine processes use to record gathered information in the InfoHub and to communicate with the Publisher. After launching FileLine and PipeLine processes, the Publisher schedules itself using HANG commands, periodically waking up to verify that its FileLine and PipeLine processes are active. During the HANG, it requires an interrupt to awaken it. Configuration processing stores Publisher specification in the following nodes: ^InfoHubConf(InfoHubID,"Publishers",PublisherID,BeginningSequenceNumber)​=EndingSequenceNumber]:​InfoDictID:​[APIDir]:​[TmpPWD]:​[TmpDBAlloc]:​[TmpDBExtend]

Example:

Publisher:SystemDict:System:::publishers/ihsyslog:1000:1000

This example configures a Publisher called System. This example is a part of the SimpleMonitor.conf configuration file in the ULFM Reference Implementation.

Related Information:

A FileLine descriptor defines aspects of monitoring a text file. For each FileLine entry in the configuration, the Publisher master process JOBs a process that reads the monitored file line by line, executing a GT.M expression (an extrinsic function call) for each line read in order to gather relevant information. The FileLine process OPENs the monitored file with the FOLLOW deviceparameter and then READs individual lines (the GT.M equivalent of tail -f).

The syntax of a FileLine descriptor is

FileLine:{InfoDictID | InfoDictName}:[FileLineName]:[FileLineID]:{PublisherID | PublisherName}:Filename:[CheckCycle]:[Timeout]:[PieceSeparator]:[PreExpr]:[InfoExpr]:[PostExpr]

InfoHub Internals

On starting up, a FileLine process executes its PreExpr, gathering any information provided, and logging an error if the data does not match the expected format, as described earlier. In the event that PreExpr does not return an error, the FileLine process OPENs the file using the FOLLOW deviceparameter, and sits on an untimed READ of the file. Every line read gets processed by InfoExpr, and the returned data placed into the InfoHub database. Triggers on ^InfoHubInfo provide asynchronous notification to Subscribers; this eliminates polling to provide the most efficient processing (all other actions, including commands to shutdown, are sent via asynchronous interrupt-driven messages). In response to a shutdown command, the FileLine process executes its PostExpr, which may provide summary information, as appropriate, before shutting down. GT.M V6.0-002 is the minimum compatible version because InfoHub uses the FOLLOW deviceparameter functionality which was first introduced in V6.0-002. For information on installing GT.M, refer to "Installing GT.M" chapter of the UNIX Administration and Operations Guide. Configuration stores FileLine specifications in the following node: ^InfoHubConfInfoHubID,"Publishers",PublisherID,BeginningSequenceNumber,​FileLineID)=​ EndingSequenceNumber]:​FileName:​InfoDictID:​FileLineName:​CheckCycle:​Timeout:​PieceSeparator]:​[PreExpr]:​[InfoExpr]:​[PostExpr] PreExpr, InfoExpr, and PostExpr always execute with $ZGBLDIR pointing to the global directory in the temporary directory. The FileLine process logs gathered information in the InfoHub database using an extended reference.

Example:

FileLine:GenericDict:OpLog::System:/var/log/messages:2:1:$char(30):PreExpr^LogFileGleaner:InfoExpr^LogFileGleaner:PostExpr^LogFileGleaner

FileLine:GenericDict:AuthLog::System:/var/log/auth.log:2:1:$char(30):PreExpr^LogFileGleaner:InfoExpr^LogFileGleaner:PostExpr^LogFileGleaner

This example configures two FileLine Gleaners, OpLog and AuthLog, as presented in the SimpleMonitor.conf configuration file in the ULFM Reference Implementation. The referenced PreExpr, InfoExpr, and PostExpr functions are implemented in "LogFileGleaner.m" (also available in the ULFM Reference Implementation), which provides a simple implementation of a gleaner that processes information from the system log.

Related Information:

A PipeLine descriptor defines the aspects of monitoring output from a process. The monitored process can be a GT.M (any version) process or any other UNIX process. The Publisher master process JOBs a PipeLine process which reads the stdout and/or stderr of the monitored process line by line, executing a GT.M expression (an extrinsic function call) for each line read in order to gather and format the per-line information. As the name indicates, a PipeLine process uses a UNIX pipe to monitor a process.

The syntax of a PipeLine Descriptor is:

PipeLine:{InfoDictID | InfoDictName}:[PipeLineName]:[PipeLineID]:{PublisherID | PublisherName}:PipeCmd:[PipeCycle]:[Timeout]:[PieceSeparator]:[PreExpr]:[InfoExpr]:[PostExpr]:

InfoHub Internals

On startup, each process for a PipeLine descriptor executes its PreExpr, gathering any information provided and logging an error if the data does not match the expected format, as described earlier. In the event PreExpr does not return an error, the PipeLine process OPENs the PipeCmd using a PIPE device and /bin/sh. The deviceparameters for the GT.M OPEN command used to start PipeCmd includes /bin/sh syntax set the environment appropriately. PipeCmd starts with environment variable changes as specified by ^InfoHubConf(InfoHubID,​"Publishers",​PublisherID,​BeginningSequenceNumber,​"EnvSetUp") for its Publisher as well as those specified in ^InfoHubConf(InfoHubID,​"EnvSetUp",​BeginningSequenceNumber) for descriptors applicable to the InfoHub itself (that is, applicable to all Publishers). In the event the PipeCmd process terminates, the PipeLine process executes PostExpr, and terminates if PostExpr returns an error. Then, if PipeCycle is -1, the PipeLine process logs in the InfoHub database the fact that it is terminating and terminates. If PipeCycle is non-negative, the PipeLine process executes PreExpr, and if the return value is zero, starts a new invocation of PipeCmd. Configuration processing stores PipeLine specifications in the following node: ^InfoHubConf(InfoHubID,"Publishers",PublisherID,BeginningSequenceNumber,​PipeLineID)=​[EndingSequenceNumber]:​PipeCycle:​InfoDictID:​PipeLineName:​PipeCmd:​Timeout:​[PieceSeparator]:​[PreExpr]:​[InfoExpr]:​[PostExpr]

Example:

PipeLine:UpTimeDict:Uptime::System:/usr/bin/uptime:15:2:$char(30):PreExpr^UptimeGleaner:InfoExpr^UptimeGleaner:PostExpr^UptimeGleaner

This example configures a PipeLine gleaner for the uptime command. This example is a part of SimpleMonitor.conf in the ULFM Reference Implementation.

Related Information:

While the InfoHub database is available to any reader for asynchronous access without a need to be identified in the Configuration File, an InfoHub can also notify a reader when InfoHub data has a characteristic specified in a Subscription (see "Defining a Subscription"). For example, the Reference Implementation SNMP sub-agent uses this technique to generate SNMP traps without an asynchronous external request or polling. While most of the InfoHub works independently of any adapte rs, in order to perform asynchronous alerts or notifications, it requires some context for the Subscriber adapters. The syntax of the Subscriber Descriptor is:

Subscriber:{InfoDictID | InfoDictName}:[SubscriberName]:[SubscriberID]

InfoHub Internals

When a Subscription detects an alert condition it queues the item (with a timestamp) under ^InfoHubInfo(InfoHubId,"Alerts") and INTRPTs all Subscribers attached to that Subscription. The Subscriber adapter process handles the queued alert item and KILLs its descendant node at its Subscriber ID; if there are no more Subscriber descendant nodes, it KILLs the queued item. Configuration stores Subscriber specifications in the following node: ^InfoHubConf(InfoHubID,"Subscribers",SubscriberID,BeginningSequenceNumber)=​[EndingSequenceNumber]:​InfoDictID:SubscriberName

Example:

Subscriber:Reporters:SNMP:404
Subscriber:Reporters:NAGIOS:503

This example configures two subscribers–SNMP and NAGIOS. This example is a part of SimpleMonitor.conf in the ULFM Reference Implementation.

Related Information:

InfoHub implements alerts and notifications with GT.M triggers on ^InfoHubInfo nodes. With one exception, a Subscription Descriptor essentially defines a trigger. A Subscription Descriptor specifying a condition of "Noinfo" differs slightly, in that it starts a Noinfo process which periodically checks for data in ^InfoHubInfo and if it finds none, places a "key:value" pair in the alert queue and sends an INTRPT to its Subscriber process(es). The syntax of the Subscriber Descriptor is:

Subscription:{InfoDictID | InfoDictName}:[SubscriptionName]: [SubscriptionID]:{InfoDictID | InfoDictName}:{InfoDictItemID | InfoDictItemName}:Condition:[Value]:[Period]:[entryref]:[SubscriberID[,...]]:[PublisherID[,...]]

InfoHub Internals

InfoHub implements other alerts with GT.M triggers on InfoHub nodes. A Subscription Descriptor other than "Noinfo" defines a trigger. The technique of using GT.M triggers to implement Subscriptions does not work for NoInfo Subscriptions because triggers are initiated by a database update, whereas NoInfo is initiated by the absence of an update. To manage polling impact, each NoInfo subscription is implemented with a process that executes a Hang for the Period at the end of which it might need to queue an alert and INTRPT one or more Subscribers. When the Hang expires, the process checks whether or not the information it is monitoring has changed. If it has, the process notes the time at which it changed, and schedules itself for the future with another Hang. If it has not changed, the process makes an appropriate entry in ^InfoHubInfo, issues the alert / notification, and schedules itself for the future with a Hang for the Period. Because the META type aggregation mechanism uses a trigger on a node that manages Subscriber notifications for its Subscription and suppresses storing of data at the trigger node, a NoInfo process has no means to track activity at a META node. Configuration processing stores Subscription specifications in the following nodes: InfoHubConf(InfoHubID,"Subscriptions",SubscriptionID,BeginningSequenceNumber)=​ [EndingSequenceNumber]:​SubscriptionInfodictID:​ItemInfoDictID:​InfoDictID:​condition:​value:​[period]:​[entryref]:​[SubscriberID[,...]]:​[PublisherID[,...]] ^InfoHubConf(InfoHubID,"NoInfo",SubscriptionID,BeginningSequenceNumber)=​

Example:

Subscription:SysSubscriptions:PatchMe:1000:UpTime:Days:<:30:::404:
Subscription:SysSubscriptions:AuthFail:2000:GenericAny:Anything:[:""""Fail"""":::503:
Subscription:SysSubscriptions:GTMError:3000:GenericAny:Anything:[:""""%GTM-E""""::::

This example configures 3 subscriptions–PatchMe, AuthFail, and GTMError. PatchMe is for Subscriber ID 404 and AuthFail is for Subscriber ID 503. GTMError is for all Subscribers. This example is a part of SimpleMonitor.conf configuration file in the ULFM Reference Implementation.

Related Information:

A gleaner is a plug-in routine invoked by an xLine (PipeLine or FileLine) process. It pairs monitored information with the relevant InfoDictItem ID(s) and returns those pairs to the xLine process for filing in the InfoHub database. Typically, you use a FileLine Gleaner for active monitoring of a text file and a PipeLine Gleaner for active monitoring of stdout and/or stderr of a process.

A gleaner must have at least one of the following extrinsic expressions (functions):

Here are some tips for writing M code for a gleaner:

Examples:

InfoHub Internals

When an xLine receives a string consisting of key-value pairs, it processes each pair individually. First, InfoHub verifies the existence of the following node to ascertain whether there is a Path in the configuration which matches the <key>: ^InfoHubConf(<InfoHubID>,<InfoHubSeqNo>,"Paths",<PublisherID>,<xLineID>,<key>) If the node exists and is of non-META type, InfoHub performs the following three updates: ^InfoHubInfo(<InfoHubID>,<PublisherID>,<PipeLineID>,<key>,<seqno>)=":<type>:<value>" ^InfoHubInfo(<InfoHubID>,"TimeToSeqNo",<PublisherID>,<PipeLineID>,<key>,<time>,<seqno>)="" ^InfoHubInfo(<InfoHubID>,"SeqNoToTime",<PublisherID>,<PipeLineID>,<key>,<seqno>)=<time> where <value> is the one corresponding to the current key; <type> is as specified in the ^InfoHubConf(...,"Paths",...) node; and <seqno> is the current sequence number for this key. "TimeToSeqNo" and "SeqNoToTime" are ancillary nodes that InfoHub uses to perform book-keeping functions. InfoHub skips the first of these updates if the value for a particular key did not change from the last time. Otherwise, InfoHub "closes" the previous update by providing the ending sequence number in the value: ^InfoHubInfo(<InfoHubID>,<PublisherID>,<xLineID>,<key>,<seqno>)="<ending seqno>:<type>:<value>" Suppose the prior value for key 9999 was .77 and the current value is .64. The following node represents the InfoHub database update at the time when key 9999 was .77: ^InfoHubInfo(11,112,1137,9999,87)=":String:.77" The following nodes represent InfoHub database updates when the value corresponding to key 9999 changes to .64. ^InfoHubInfo(11,112,1137,9999,87)="88:String:.77" ^InfoHubInfo(11,112,1137,9999,88)=":String:.64" Here 88 is the ending sequence number of the prior value. The change for key 9999's corresponding value to .64 also adds the following nodes: ^InfoHubInfo(11,"SeqNoToTime",112,1137,9999,88)="1385382167772674" ^InfoHubInfo(11,"TimeToSeqNo",112,1137,9999,1385382167772674,88)="" In case of META nodes, it performs the following update in addition to the updates to the node causing the META action: ^InfoHubInfo(<InfoHubID>,"Alerts",<PublisherID>",<xLineID>,<key>)="<time>:<SubscriptionID>:info:ref:<last non-meta path for this event >"

Related Information:

This appendix provides information about the FIS GT.M SNMP Plugin. Comfort with SNMP concepts, if not outright familiarity, will help you understand the concepts in this Appendix. If you do not have a working knowledge of SNMP concepts, the following tutorials, as well as many others may be useful:

The FIS GT.M SNMP Plugin is a ready-to-use subagent that makes information from InfoHub available for monitoring via SNMP. It uses the Internet standard (RFC-2741) AgentX protocol to communicate with an SNMP Master Agent. Because FIS tests the SNMP Plugin with Net-SNMP ( http://www.net-snmp.org) as the SNMP Master Agent, Net-SNMP is Supported and other SNMP Master Agents are Supportable but not Supported.

The SNMP Plugin is a Reporting Adaptor that has both Query and Subscriber relationships with InfoHub. The following diagram illustrates how the SNMP Plugin interacts with InfoHub to handle requests for data retrieval and notifications from an SNMP environment.

A data retrieval request initiated from CLI applications such as snmpget/snmpgetnext goes to the Master SNMP Agent using the SNMP protocol. The Master SNMP Agent uses the AgentX protocol to communicate the request to the SNMP Plugin which uses its Query relationship with InfoHub to retrieve the requested data and sends it to back to the Master SNMP Agent. CLI applications such as snmptrapd set up a listener for receiving notifications. When a configured condition occurs on InfoHub, the SNMP Plugin uses its Subscriber relationship to obtain notification details and sends them to the SNMP Master Agent which then sends it to the listener waiting for notifications.

In an SNMP environment, data values are organized in a tree structure called an MIB (Management Information Base). The MIB hierarchy starts with a nameless root and extends to specific areas of information. Typically, an organization has several MIB modules connected hierarchically with each other. Each object is addressed using a fully qualified OID starting from the nameless root. An OID can be represented in numeric or mnemonic form. A fully qualified OID in numeric form looks like:

.1.3.6.1.4.1.16830.6.2.1.2.1.1482247793.1.314.0

The same OID in mnemonic form might looks like:

.iso.org.dod.internet.private.enterprises.fis.profile.gtm.infohub.profile1curr.uatenv1curr.syslogcurr.SyslogAllcurr.0

The following diagram illustrates the fully qualified OID .1.3.6.1.4.1.16830.6.2.1.2.1.1482247793.1.314.0 in an MIB that contains an InfoHub MIB module sub-tree. The numeric forms are enclosed in brackets ().

The non-white boxes represent the OID sub-subtree in the InfoHub MIB module. In the illustration, the InfoHub MIB module hierarchy starts from 16830.6.2 where:

InfoHub maintains the current and historic data values of monitored elements in the form of Views. InfoHub provides four Views – curr, hist, time, and snum. Views are different ways of looking at the current and historic data values of an object. A View is like a filter applied on an object to narrow down to only those data values that you want to see from an InfoHub. Every object under an InfoHub has four stems each representing a View. All objects under a stem have the same View Name as the suffix. The last (leaf) node of each stem is the monitored object for which you need to specify a parameter called an Object Index.

The objective of stemming of the same object into four Views is to comply with the namespace restriction that all individual pieces of an OID in an MIB to be unique. The non-white boxes in the OID illustrate a fully qualified mnemonic OID of the most recent (curr) object stored on the system log and the curved connectors denote stemming into Views. Notice how each object under .profilecurr has the curr suffix and the leaf node requires specifying an Object Index^[2].

To obtain the current or historic data of any monitored element, you need to perform a GET request (refer to "Performing a GET Request") for either the fully qualified OID as illustrated in the OID diagram or just the leaf node (for example, .infohubprofile1uatenv1SyslogAllcurr.0).

The four Views are as follows:

Related Information:

To install the SNMP Plugin, perform the following steps:

Related Information:

Using the same environment setup described for the InfoHub, you can operate the SNMP Plugin from the shell and from within GT.M. To operate the SNMP Plugin from the shell, the general command is:

$gtm_dist/mumps -run IHsnmp --action=<Action> --plugin=PluginId 

To operate the SNMP Plugin from within GT.M, invoke the action entry point in the IHsnmp API:

action^IHsnmp("<Action>","PluginID")

Related Information:

To start the SNMP Plugin from the shell, execute a command like:

$ $gtm_dist/mumps -run IHsnmp --action=start --plugin=PluginId 

To start the SNMP Plugin from within GT.M, invoke the action entry point in the IHsnmp API and pass "start" as the first parameter.

action^IHsnmp("start","Pluginid")

Example:

$gtm_dist/mumps -run IHsnmp --action=start --plugin=5 

GTM> do action^IHsnmp("start","5")

Both examples start the SNMP Plugin.

Related Information:

To check the status of the SNMP Plugin from the shell, execute a command like:

$ $gtm_dist/mumps -run IHsnmp --action=status --plugin=PluginId

This command displays a summary of the current status of the SNMP Plugin and whether it is connected to the Master SNMP Agent.

To check the status of an SNMP Plugin from within GT.M, invoke the action entry points in the IHsnmp API and pass "status" as the first parameter:

action^IHsnmp("status","Pluginid")

Example:

$gtm_dist/mumps -run IHsnmp --action=status --plugin=5

GTM>do action^IHsnmp("status","5")

Both examples display the status of the SNMP Plugin.

To shut down the SNMP Plugin from the shell, execute a command like:

$gtm_dist/mumps -run IHsnmp --action=shutdown --plugin=PluginId

This command effects a clean shuts down of the SNMP Plugin.

To shutdown an SNMP Plugin from within GT.M, invoke the action entry points in the IHsnmp API and pass "shutdown" as the first parameter:

action^IHsnmp("shutdown","5")

Example:

$gtm_dist/mumps -run IHsnmp --action=shutdown --plugin=5 

GTM>do action^IHsnmp("shutdown","5")

Both examples shutdown the SNMP Plugin.

The snmpget CLI application performs GET requests from an InfoHub SNMP module. The snmpget command retrieves data from monitored objects according to the specified View. A typical syntax of the snmpget command for the SNMP Plugin is:

snmpget [<snmpcmd_common_options>] -v2c -c <communitystring> -Oa [-m <MIBModule(s)>] OID [OID]...

For more information on the snmpget command options, refer to the man page of the snmpget command.

Examples:

With curr View:

$ snmpget -v2c -c <communitystring> -Oa -m INFOHUB-1-MIB localhost INFOHUB-1-MIB::infohubtestMyBoxSyslogDBFILEXTcurr.0

This example retrieves the current data value of the object infohubtestMyBoxSyslogDBFILEXTcurr.0 from the InfoHub MIB module.

With hist View:

$ snmpget -v2c -c <communitystring> -Oa -m INFOHUB-1-MIB localhost INFOHUB-1-MIB::infohubtestMyBoxSyslogDBFILEXThist.1

This example retrieves the data value of the object prior to the current value. Object Index 1 specifies the value that was set for the infohubtestMyBoxSyslogDBFILEXT object before the current value.

$ snmpget -v2c -c <communitystring> -Oa -m INFOHUB-1-MIB localhost INFOHUB-1-MIB::infohubtestMyBoxSyslogDBFILEXThist.0

This example retrieves the current data value of the object. Note that Object Index 0 for the hist View is the same as curr.0 and specifies the current data value.

With time View:

The time View is only available with snmpgetnext. Because time is specified in seconds since the last UNIX epoch and InfoHub time granularity is much less than one second, snmpget is not able to determine the exact time. For more information see "Performing a GETNEXT request".

With snum View:

$ snmpget -v2c -c <communitystring> -Oa -m INFOHUB-1-MIB localhost INFOHUB-1-MIB::infohubtestMyBoxSyslogGTMFatalsnum.1011912

This example retrieves the data value of a GT.M Fatal error stored on InfoHub with sequence number 1011912. In most cases, you would use the snum View in conjunction with the time View which returns the sequence number of the monitored object's mnemonic OID.

AIX Notes

On AIX, you need to explicitly specify the port number for the NET-SNMP Agent since another port was assigned during installation (see the agentAddress line in snmpd.conf). A sample snmpget command might look like: snmpget -v2c -c <communitystring> -Oa -m INFOHUB-1-MIB 127.0.0.1:9716 INFOHUB-1-MIB::infohubtestMyBoxSyslogAllcurr.0 or: snmpget -v2c -c <communitystring> -Oa -m INFOHUB-1-MIB 127.0.0.1:1610 INFOHUB-1-MIB::infohubtestMyBoxSyslogAllcurr.0

Related Information:

The snmpgetnext CLI application performs GETNEXT requests from an InfoHub SNMP module. The snmpgetnext command retrieves data that lies next to the specified objects in the order specified by the View. A typical syntax of the snmpgetnext command in the SNMP Plugin is:

snmpgetnext [<snmpcmd_common_options>] -v2c -Oa -c <communitystring> [-m <MIBModule(s)>] OID [OID]...

For more information on the snmpgetnext command options, refer to the man page of the snmpgetnext command.

Examples:

With curr View:

$ snmpgetnext -v2c -c <communitystring> -Oa -m INFOHUB-1-MIB localhost INFOHUB-1-MIB::infohubtestMyBoxSyslogDBFILEXTcurr.0

This example retrieves the data value of the object that lies immediately after infohubtestMyBoxSyslogDBFILEXT in the InfoHub MIB module.

With hist View:

$ snmpgetnext -v2c -c <communitystring> -Oa -m INFOHUB-1-MIB localhost INFOHUB-1-MIB::infohubtestMyBoxSyslogDBFILEXThist.1

This example retrieves the predecessor (that is, hist.2) of the hist.1 value of the object infohubtestMyBoxSyslogDBFILEXThist.

With time View:

$ snmpgetnext -v2c -c <communitystring> -Oa -m INFOHUB-1-MIB localhost INFOHUB-1-MIB::infohubtestMyBoxSyslogGTMFataltime.`date --date "10/09/2013 19:05:00" +%s`
INFOHUB-1-MIB::infohubtestInfoHubOID = OID: INFOHUB-1-MIB::infohubtestMyBoxSyslogGTMFatalsnum.19

This example retrieves the mnemonic OID and the sequence number of the first fatal GT.M error that was stored on Infohub after 10/09/2013 14:26:00. This View is useful in situations where you need to investigate incidents occurring around a specified time.

With snum View:

$ snmpgetnext -v2c -c <communitystring> -Oa -m INFOHUB-1-MIB localhost INFOHUB-1-MIB::infohubtestMyBoxSyslogGTMFatalsnum.18
INFOHUB-1-MIB::infohubtestMyBoxSyslogGTMFatalsnum.19 = STRING: "Oct  9 19:06:07 gtmnode20 GTM-SRCSRVR-INSTANCE1[14796]: %GTM-F-FORCEDHALT, Image HALTed by MUPIP STOP -- generated from 0x0000000000000000."

This example uses the sequence number (snum.18) returned with the time View example to retrieve the data value of the next GT.M Fatal error stored on InfoHub immediately after sequence number 18. In most cases, you would use the snum View in conjunction with the time View which returns the sequence number of the monitored object's mnemonic OID.

You can also use the snmpbulkget command to display the next 10 data values of objects stored in InfoHub after sequence number 18.

$ snmpbulkget -v2c -c <communitystring> -Oa -m INFOHUB-1-MIB localhost INFOHUB-1-MIB::infohubtestMyBoxSyslogGTMFatalsnum.18

You can also specify -Cr20 to retrieve the next 20 data values of objects stored in InfoHub after sequence number 18.

AIX Notes

On AIX, you need to explicitly specify the port number for the NET-SNMP Agent since another port was assigned during installation (see the agentAddress line in snmpd.conf). A sample snmpgetnext command might look like: snmpgetnext -v2c -c <communitystring> -Oa -m INFOHUB-1-MIB 127.0.0.1:9716 INFOHUB-1-MIB::infohubtestMyBoxSyslogAllcurr.0 or: snmpgetnext -v2c -c <communitystring> -Oa -m INFOHUB-1-MIB 127.0.0.1:1610 INFOHUB-1-MIB::infohubtestMyBoxSyslogAllcurr.0

Related Information:

The snmptranslate CLI application can be used to explore/traverse the OIDs in your InfoHub MIB. The snmptranslate command translates one or more OIDs in numeric form to textual form or vice versa. A typical syntax of the snmptranslate command in the SNMP Plugin is:

snmptranslate [<snmpcmd_common_options>] -m <MIBModule(s)> {-Tp|-Td}  {-Oa|-On} [-IR] OID [OID]...

Example:

Traversing an OID using the Tree View:

$ snmptranslate -m INFOHUB-1-MIB -Tp | less
+--gtm(2)
|
 +--infohub(1)
  |
  +--profile1(2)
     |
     +--profile1curr(1)
     |  |
     |  +--profile1uatenv1curr(641977582)
     |     |
     |     +--profile1uatenv1Syslogcurr(1)
     |     |  |
     |     |  +-- -R-- String    profile1uatenv1SyslogGTMcurr(159)
     |     |  +-- -R-- String    profile1uatenv1SyslogGTMFatalcurr(265)
     |     |  +-- -R-- String    profile1uatenv1SyslogAllcurr(314)
     |     |  +-- -R-- String    profile1uatenv1SyslogGTMFailcurr(323)
     |     |  +-- -R-- String    profile1uatenv1SyslogGTMsecshrcurr(358)
     |     |  +-- -R-- String    profile1uatenv1SyslogGTMKillcurr(979)
     |     |  +-- -R-- String    profile1uatenv1SyslogJNLACCESScurr(150372826)
     |     |  +-- -R-- String    profile1uatenv1SyslogGBLOFLOWcurr(150372882)
     |     |  +-- -R-- String    profile1uatenv1SyslogGTMSECSHRDMNSTARTEDcurr(150373907)
     |     |  +-- -R-- String    profile1uatenv1SyslogJNLSPACELOWcurr(150375771)
     |     |  +-- -R-- String    profile1uatenv1SyslogGTMSECSHRSRVFcurr(150376298)
     |     |  +-- -R-- String    profile1uatenv1SyslogGTMASSERTcurr(150376532)
     |     |  +-- -R-- String    profile1uatenv1SyslogTPRESTARTcurr(150376595)
     |     |  +-- -R-- String    profile1uatenv1SyslogMUTEXLCKALERTcurr(150376920)
     |     |  +-- -R-- String    profile1uatenv1SyslogGTMSECSHRcurr(150376938)
     |     |  +-- -R-- String    profile1uatenv1SyslogDBFILEXTcurr(150377491)
     |     |  +-- -R-- String    profile1uatenv1SyslogJNLFSYNCERRcurr(150377498)
     |     |  +-- -R-- String    profile1uatenv1SyslogDBFSYNCERRcurr(150378082)
     |     |  +-- -R-- String    profile1uatenv1SyslogNOSPACECREcurr(150378762)
     |     |  +-- -R-- String    profile1uatenv1SyslogOUTOFSPACEcurr(150378788)
     |     |  +-- -R-- String    profile1uatenv1SyslogTPNOTACIDcurr(150378867)
     |     |  +-- -R-- String    profile1uatenv1SyslogDBDANGERcurr(150378976)
     |     |  +-- -R-- String    profile1uatenv1SyslogREQ2RESUMEcurr(150379035)
     |     |  +-- -R-- String    profile1uatenv1SyslogSUSPENDINGcurr(150379139)
     |     |  +-- -R-- String    profile1uatenv1SyslogSTUCKACTcurr(150381523)
     |     |  +-- -R-- String    profile1uatenv1SyslogGTMASSERT2curr(150382164)
     |     |  +-- -R-- String    profile1uatenv1SyslogREPLINSTFROZENcurr(150382650)
     |     |  +-- -R-- String    profile1uatenv1SyslogREPLINSTUNFROZENcurr(150382667)
     |     |
     |     +--profile1uatenv1RegReplFetchcurr(583368784)

The leaf nodes under profile1uatenv1Syslogcurr(1) start from profile1uatenv1SyslogGTMcurr(159) up to profile1uatenv1SyslogREPLINSTUNFROZENcurr(150382667). As explained in the "Performing an SNMP GET Request", a GET request for profile1uatenv1SyslogDBFILEXTcurr.0 produces the current value.

The InfoHub Configuration File entries for profile1uatenv1SyslogDBFILEXTcurr.0 can be as follows:

These entries are tied together in the InfoHub Configuration File as follows:

InfoHub:profile1:2
Publisher:Instances:uatenv1:::infogleaner:1000:1000
FileLine:Xline:Syslog:1:uatenv1:/var/log/messages:2:1:​$char(30):​PreExpr^FLGleaner:​InfoExpr^FLGleaner
InfoDict:Instances
InfoDictItem:Instances:uatenv1:::
InfoDict:SyslogParent:1:Instances 
InfoDictItem:SyslogParent:Syslog:1:::
// All syslog messages
InfoDict:SyslogAll::SyslogParent
InfoDictItem:SyslogAll:All:314:::All syslog messages
// All GT.M syslog messages
InfoDict:SyslogGTM::SyslogParent
InfoDictItem:SyslogGTM:GTM:159:::All GT.M syslog messages
// All fatal syslog messages
InfoDict:SyslogFatal::SyslogParent
InfoDictItem:SyslogFatal:GTMFatal:265:::All fatal syslog messages
// Individual syslog message paths
InfoDict:SyslogIndividual::SyslogParent
InfoDictItem:SyslogIndividual:JNLACCESS:150372826::String
InfoDictItem:SyslogIndividual:GBLOFLOW:150372882::String
InfoDictItem:SyslogIndividual:GTMSECSHRDMNSTARTED:150373907::String
InfoDictItem:SyslogIndividual:JNLSPACELOW:150375771::String
InfoDictItem:SyslogIndividual:GTMSECSHRSRVF:150376298::String
InfoDictItem:SyslogIndividual:GTMASSERT:150376532::String
InfoDictItem:SyslogIndividual:TPRESTART:150376595::String
InfoDictItem:SyslogIndividual:MUTEXLCKALERT:150376920::String
InfoDictItem:SyslogIndividual:GTMSECSHR:150376938::String
InfoDictItem:SyslogIndividual:DBFILEXT:150377491::String

Displaying full details of an OID:

$ snmptranslate -m INFOHUB-1-MIB -Td -IR -Oa profile1uatenv1SyslogGTMcurr
INFOHUB-1-MIB::profile1uatenv1SyslogGTMcurr
profile1uatenv1SyslogGTMcurr OBJECT-TYPE
  -- FROM       INFOHUB-1-MIB
  SYNTAX        OCTET STRING
  MAX-ACCESS    read-only
  STATUS        current
  DESCRIPTION   "All GT.M syslog messages"
::= { iso(1) org(3) dod(6) internet(1) private(4) enterprises(1) fis(16830) profile(6) gtm(2) infohub(1) profile1(2) profile1curr(1) profile1uatenv1curr(641977582) profile1uatenv1Syslogcurr(1) 159 }

This example displays all details associated with profile1uatenv1SyslogGTMcurr.

Displaying an OID in numeric form:

$ snmptranslate -m INFOHUB-1-MIB -IR -On profile1uatenv1SyslogGTMcurr
.1.3.6.1.4.1.16830.6.2.1.2.1.641977582.1.159

This example displays profile1uatenv1SyslogGTMcurr in numeric form.

Related Information:

The snmptrapd CLI application sets up a listener for SNMP notifications (TRAP or INFORM) from the Master SNMP Agent. The SNMP Plugin uses its Subscriber relationship with InfoHub to obtain notification details and sends them to the SNMP Master Agent. Then, snmptrapd receives those notifications in the form of a TRAP or an INFORM. INFORM differs from a TRAP in that it requires acknowledgement message, while the TRAP does not. Because snmptrapd, by default, listens for SNMP notifications on the special privilege UDP port 162 on IPv4 interfaces, you need to run the snmptrapd command as root. At any given time, only one snmptrapd can receive notifications but you can set up one or more listening addresses (refer to the man page entry for snmpd for more information). A typical syntax of the snmptrapd command in the SNMP Plugin is:

snmptrapd[<snmpcmd_common_options>] -m <MIBModule(s)>
-Lo -f -t -Oa

For more information on the snmptrapd command options, refer to the man page of the snmptrapd command.

Examples:

$ sudo snmptrapd -m INFOHUB-1-MIB -Lo -f -t -Oa
SNMPv2-SMI::snmpModules.1.1.4.1.0 = OID: INFOHUB-1-MIB::infohubtestMyBoxSyslogDBFILEXTcurr.0
2013-10-23 03:34:10 localhost.localdomain [UDP: [127.0.0.1]:54501->[127.0.0.1]]:
SNMPv2-SMI::mib-2.1.3.0 = Timeticks: (118883997) 13 days, 18:13:59.97   
SNMPv2-SMI::snmpModules.1.1.4.1.0 = OID: INFOHUB-1-MIB::infohubtestMyBoxSyslogDBFILEXTcurr.0

This example sets up a listener for notifications coming from InfoHub. -Lo specifies that the messages are logged on standard output. The DBFILEXT notification that arrives on snmptrapd is from a Noinfo subscription called DBFILEXT which InfoHub raised due to lack of activity for the DBFILEXT message.

	AIX Notes
	On AIX, use the NET-SNMP executable to set up a listener for notifications coming from InfoHub. A sample snmptrapd command might look like: sudo /usr/local/sbin/snmptrapd -m INFOHUB-1-MIB -Oa -Lo -f -t

Related Information:

Your InfoHub distribution comes with a GT.M Monitoring Reference Implementation that can monitor GT.M messages in the system log and GT.M databases V6.0-002 and above. It is possible to monitor any GT.M version prior to V6.0-002 with some modifications. Modifications are necessary because the format and content of file header listings, messages, and gleaning techniques vary from release to release. For example, to monitor an earlier release which lacks the $ZPEEK() function, you would have to use other techniques, such as parsing DSE output for example, with a program such as the %DSEWRAP utility.

The following table contains information about the files that are used with the GT.M Monitoring Reference Implementation. These files reside in an InfoHub installation. You can view the configuration files from the samples subdirectory of your InfoHub distribution or download them from the links in the table. You can also click to download all files of the GT.M Monitoring Reference Implementation together or open directly from http://tinco.pair.com/bhaskar/gtm/applications/infohub/GTM_Monitoring_RI.zip.

The following illustration shows how the GT.M Monitoring Reference Implementation sets up an InfoHub framework for monitoring your GT.M database.

This section explains the operation of the Syslog FileLine Gleaner from the GT.M Monitoring Reference Implementation and discusses the caveats of gleaning the syslog messages on a system with multiple GT.M versions in use.

The following summarizes categorization used by the Syslog Gleaner:

When processing GT.M messages, the Syslog Gleaner utilizes preconstructed parsing clues (primarily for argument extraction) specific to the GT.M version that the gleaner is running on. Because old messages occasionally get removed or reworded in new releases of GT.M, the Syslog Gleaner may sometimes be unable to recognize or parse a particular message for the purpose of reporting it in an appropriate InfoDictItem. This is a possibility when a system has multiple database instances with different GT.M versions configured to report their messages in the same syslog file. InfoHub logs every occurrence of unparseable or unrecognizable message in the syslog as well as in the error InfoDictItem, if one is configured.

Depending on the OS and configuration of the syslog utility, certain messages may never reach the log file due to the use of load-induced skipping or repetition-induced compression techniques; such messages are thus unavailable for gleaning. For example, the popular rsyslog daemon allows the reduction of repeated messages via the $RepeatedMsgReduction configuration directive. When in effect, this mode replaces repeated content with a message like "last line repeated n times." Because such messages only qualify in the most general category (All Messages InfoDictItem), use message reduction with the FileLine Syslog Gleaner after considering its impact. Factors to contemplate in this regard are any rate based Subscriptions and Subscriptions to critical messages such as the ones from gtmsecshr or having fatal severity.

The Uptime and Log File Monitoring (ULFM) Reference Implementation monitors:

The following table contains information about all files that are a part of the ULFM Reference Implementation. You can view these files from the configs subdirectory of your InfoHub distribution where they reside or download them from the links in the table. You can also click to download all files of the ULFM Reference Implementation together or open directly from http://tinco.pair.com/bhaskar/gtm/applications/infohub/ULFM_RI.zip.

To install the ULFM Reference Implementation, perform the following steps:

The following illustration shows how the ULFM Reference Implementation sets up an InfoHub framework for monitoring.

Related Information:

IHABSPATHREQ, FileName must have an absolute path

InfoBub Error: This error occurs configuration processing detects a FileLine descriptor entry where the monitored FileName does not have an absolute path.

Action: Always specify an absolute path for the monitored FileName.

The InfoHub dddd is active, operation start is not permitted; yyyy

InfoHub Error: This error indicates the the InfoHub named dddd is active and cannot be started. yyyy lists the two likely actions.

Action: Please follow the suggestion in yyyy.

IHASSERT, Assert failed - dddd

InfoHub Error: This error indicates an out of design condition; dddd contains additional context.

Action: Please contact your support channel.

IHAXNFAIL, An InfoHub service [Publishers|NoInfo|FileLine|PipeLine], ID: cccc, failed to [start|restart|shutdown]

InfoHub Warning : The parent's request to start, restart or shutdown the target service failed.

Action: This message indicates a failure for a service to respond to a request. Consult the service's log file to more information.

IHBADALLOC, TmpDBAlloc must be within the range accepted by GDE

InfoHub Error: This error occurs when configuration processing detects an invalid value for TmpDBAlloc (the initial allocation of the database in the temporary directory–with a predetermined block size of 4KiB) in the Publisher descriptor entry.

Action: Specify the correct TmpDBAlloc value based on the requirements of the gleaners managed by this Publisher.

IHBADAPIDIR, APIDir must have a valid path

InfoHub Error: This error occurs when configuration processing detects an invalid value for APIDir which specifies the path to the routines that contain the gleaner plug-in routines. Note the configuration checks for the syntax rather than access to this directory.

Action: Specify the correct APIDir value based on the requirements of the gleaners managed by this Publisher.

IHBADCOND4PER, When there's a Period, the Condition must contain < or >

InfoHub Error: This error occurs when configuration processing encounters a Subscription with Period and Condition that does that does not contain "<" or ">".

Action: Ensure that a Subscription descriptor specifying a Period also specifies a valid Condition for a rate.

IHBADCYCLE, Cycle must have a numeric value in seconds

InfoHub Error: This error occurs when configuration processing detects a non-numeric value for CheckCycle in the FileLine descriptor entry.

Action: Always specify a numeric value of seconds for CheckCycle. Note that a value of 0 (the default) for CheckCycle means InfoHub should not check for a newer file.

IHBADDBPATH, The global directory probably does not use a fully qualified path for the database file

InfoHub Fatal: The InfoHub can't start because it needs an absolute path to the InfoHub database.

Action: Ensure the global directory used for the InfoHub specifies an absolute path to the database file holding the InfoHub globals..

IHBADDESCTYPE, Descriptors must have a valid type in the first colon (:) delimited piece

InfoHub Error: This error occurs when configuration processing detects an unrecognized descriptor.

Action: Select a valid descriptor. Valid descriptors include InfoHub, Include, Env, InfoDict, InfoDictItem, Publisher, FileLine, PipeLine, Subscriber, and Subscription.

IHBADDPSEP, PieceSeparator must be a single character other than ASCII <LF> ($CHAR(10)) or 'VAR' for gleaner defined; chose a character that is not in the data

InfoHub Error: This error occurs when configuration processing detects that the specified PieceSeparator is not a single character or the case-insensitive keyword "var" in the FileLine descriptor entry.

Action: Select a valid single character PieceSeparator; colon (:) must be specified by omitting the PieceSeparator and accepting the default.

IHBADENV, EnvVarName must be a legal POSIX name, and the operation provoke no error from /bin/sh

InfoHub Error: This error occurs when configuration processing encounters an Env descriptor having EnvVarName that is not a legal POSIX name. This error also occurs when the shell reports an error when configuration processing passes each EnvVarName value to a shell invocation.

Action: Ensure that the EnvVarName is a legal POSIX name. A legal POSIX name consists solely of ASCII letters, digits, and underscores ("_"), of which the first character must not be a digit. If the error occurs from the shell, simulate the error condition in the shell environment and rectify the error.

IHBADEXPR, PreExpr, IntExpr and PostExpr must be legal M labelref

Publisher Error: This error occurs when the Publisher detects that the PreExpr, InfoExpr, or PostExpr return a value which is not in the expected format. The Publisher logs this error and stops processing the FileLine.

Action: Ensure that the FileLine Gleaner's PreExpr and PostExpr return an empty string and InfoExpr return values in multiples of <key>:<value>.

IHBADEXT, TmpDBExtend must be within the range accepted by GDE

InfoHub Error: This error occurs when configuration processing detects an invalid value for TmpDBExtend (the extension count) in the Publisher descriptor entry.

Action: Specify the correct TmpDBExtend value based on the expected storage patterns of the gleaners managed by this Publisher.

IHBADFILE, The configuration or an Include File must be found and readable

InfoHub Error: This error occurs when the Configuration File or an Include File is not readable. This error may be accompanied by additional error context.

Action: Check that the file exists and the authorizations are appropriate; verify the file is correct, for example, by accessing it using your editor.

IHBADNAME, Names must start with '%' or an alpha, contain only alphanumeric characters and not be more than 32 characters long

InfoHub Error: This error occurs when configuration processing detects a Name in an invalid format.

Action: Ensure that Names (such as a PublisherName) start with an ASCII alphabetic character, followed by zero to 31 ASCII alphanumeric characters. Except where otherwise explicitly stated, InfoHub uses case-sensitive Names. FIS recommends short Names for SNMP MIB generation, which concatenates names.

IHBADPUBLISHER, The Publisher must have a PublisherID defined in the associated InfoDict

InfoHub Error: This error occurs when configuration processing detects a PublisherID that is not defined..

Action: Ensure that you define the Publisher when using it in a Subscription descriptor.

IHBADSTOREDCONFIG, The InfoHubID identifies a configuration with no Name

InfoHub Error: This error occurs when configuration processing finds that the InfoHubID corresponds to an InfoHub with no Name.

Action: This indicates the InfoHub configuration has an invalid state, possibly because it was improperly manipulated. You need to configure with a different InfoHubID, or remove the damaged InfoHub and its configuration, possibly with a Purge action.

IHBADSUBSCRIBER, The Subscriber must have a SubscriberID defined in the associated InfoDict

InfoHub Error: This error occurs when configuration processing detects a SubscriberID that is not defined.

Action: Ensure that you define the Subscriber when using it in a Subscription descriptor.

IHBADTEMPPWD, TempPWD must have a valid path

InfoHub Error: This error occurs when configuration processing detects an invalid value for TempPWD which specifies the path to the directory the InfoHub is to create for working space. Note the configuration checks for the syntax rather than access to this path.

Action: Specify the correct TempPWD value based on the requirements of this Publisher.

IHBADTIMEOUT, Timout must have a numeric value of seconds

InfoHub Error: This error occurs when configuration processing detects a non-numeric value for TimeOut in an XLine descriptor entry.

Action: Always specify a numeric value of seconds for TimeOut.

IHCFGNOTACTIVE, The configuration for ssss has been deactivated

InfoHub Warning: The ssss NOINFO Subscription has been deactivated by a configuration update

Action: Verify that the configuration change was appropriate.

IHCIRCDICT, InfoDict/InfoDictItem path parent-child relationships must be free of cycles

InfoHub Error: This error occurs when configuration processing detects a circular path in InfoDict parent references. Usually this error arises from having the same item name in different InfoDict domains which have the same parent.

Action: Ensure that there are no circular paths in the InfoDict Domain / Item hierarchy defined in your configuration file.

IHCLIBADOPT, yyyy

yyyy describes the missing or incorrect command line option.

Action: Fix the command line options to match operation.

IHCONFLOCKED, Only one process can configure a given InfoHub at the same time

InfoHub Error: This error occurs when configuration processing detects another process holding a lock on an InfoHub configuration for more than 30 seconds.

Action: Ensure that only one process at a time attempts to configure an given InfoHub.

IHDEADDICT, Every InfoDict must have at least one InfoDictItem

InfoHub Error: This error occurs when configuration processing detects an empty InfoDict.

Action: Ensure complete and correct declarations for InfoDictIDs and InfoDictItems; an InfoDict must have at least one InfoDictItem.

IHDEBUG, dddd

InfoHub Information: This message appears when InfoHub is run in DEBUG mode and dddd is the debug output.

Action: Take guidance and knowledge.

IHDICTNAMEREQ, An InfoDict requires a Name

InfoHub Error: This error occurs when configuration processing finds no InfoDict Name.

Action: Ensure that you properly define each InfoDict with a Name and appropriate ID in your configuration file.

IHDUPDICT, Multiple InfoDicts cannot share the same Name or ID

InfoHub Error: This error occurs when configuration processing detects an attempt to map multiple InfoDictIDs to one Name.

Action: Ensure that only one InfoDictID maps to one Name.

IHDUPFILELINE, A FileLine for a Publisher, cannot share the same Name or ID

InfoHub Error: This error occurs when configuration processing detects more than one FileLine for the same Publisher has the same Name or ID.

Action: Always specify a unique ID and Name for a FileLine associated with a Publisher.

IHDUPITEM, Multiple InfoDictItems in an InfoDict cannot share the same Name or ID

InfoHub Error: This error occurs when configuration processing detects an attempt to map multiple InfoDictItemIDs to one Name.

Action: Ensure that each InfoDictItem maps to one InfoDictItemName in your configuration file.

IHDUPPIPELINE, A PipeLine for a Publisher, cannot share the same Name or ID

InfoHub Error: This error occurs when configuration processing detects more than one PipeLine for the same Publisher has the same Name or ID.

Action: Always specify a unique ID and Name for a PipeLine associated with a Publisher.

IHDUPPUBLISHER, Multiple Publishers cannot share the same Name or ID

InfoHub Error: This error occurs when configuration processing detects an attempt to map multiple PublisherIDs to one Name.

Action: Ensure that an InfoDict domain exists for a PublisherName and only one Publisher maps to one PublisherName.

IHDUPSUBSCRBR, Multiple Subscribers associated with a single Publisher cannot share the same Name or ID

InfoHub Error: This error occurs when configuration processing detects the same ID for more than one Subscriber associated with a Publisher.

Action: Ensure that you specify a unique ID for each Subscriber.

IHDUPSUBSCRPTN, An InfoDictItem can have only one Subscription

InfoHub Error: Although a Subscription can have multiple Publishers and Subscribers, an InfoDictItem can only have one Subscription.

Action: If you need multiple conditions on a single item, consider a custom trigger.

IHENVNOFF, Form Feed (<FF>) characters are not permitted in ENV decriptors

InfoHub Error: This error occurs when configuration processing detects that the Env Descriptor contains one or more form feed (<FF>) characters, which cannot be used in Env descriptors. (/bin/sh).

Action: reformulate the ENV descriptor to avoid the FF character(s) or switch to using a script invocation.

IHENVPLACE, ENV descriptors must follow either the InfoHub descriptor or a Publisher descriptor

InfoHub Error: This error occurs when configuration processing encounters an environment descriptor that does not follow the InfoHub descriptor or a Publisher descriptor.

Action: Ensure that your Env descriptors always follow the InfoHub descriptor or a Publisher descriptor.

IHETRAPINVOK, Error trap invoked

InfoHub Fatal: An InfoHub component encountered a fatal condition.

Action: Look for associated message and for a context file produced by the error for additional information.

IHEXTRADELIM, The descriptor contains more colon (:) delimiters than the descriptor type supports

InfoHub Error: This error occurs when configuration processing detects too many colon (:) delimiters in a descriptor.

Action: Check the syntax of the descriptor. Ensure that your descriptor uses colon delimiters only in appropriate places.

IHFLMISSDTLS, An InfoHub FileLine process might have missed records due to log file rotation (current inode:dddd, prior inode:pppp

InfoHub Warning: An InfoHub monitors files continuously but only checks whether the file has been replaced periodically. If the period of it checking for a new file is configured to be long, but configuration or events cause the file to be replaced on a shorter cycle, the InfoHub may miss the records in a file which is quickly renamed or replaced. The message lists the inodes of the current, dddd, and prior, pppp, file.

Action: Ensure that the FileLine Cycle is short enough to minimize or preclude this issue.

IHFLNOROTATE, Could not verify the status of file ffff. No rotation performed

InfoHub Warning: The currently monitor file seems to no longer be available but no new file of the same name seems to be available either

Action: Investigate why the monitored file appears unavailable to the InfoHub monitoring process - does it exist, are the authorizations, including those on the path appropriate? Adjust as appropriate.

IHGETDTTIMSTRFAIL, .msg entry

InfoHub Warning: This error indicates a problem with the GT.M POSIX plugin.

Action: Please check the GT.M POSIX plugin configuration.

IHGETTIMEFAIL, Could not obtain time; external call to gtmposix.gettimeofday() failed; path to gtmposix: pppp

InfoHub Error: This error indicates that the call to the GT.M POSIX plugin function gettimeofday (man gettimeofday for more information) failed. pppp is the path to the External Call Table file.

Action: Please check the GT.M POSIX plugin configuration.

IHGLNRERR, Gleaner error report: eeee

InfoHub Error: a Gleaner encountered an error described by eeee

Action: Analyze and address the issue described by eeee

IHHUBDESCREQ, A configuration requires a way identify an InfoHub - there is no explicit identification and defaulting did not find an existing InfoHub

InfoHub Error: If the InfoHub Descriptor is not specified, configuration processing uses the current value of gtmgbldir to see if the current global directory was previously associated with an InfoHubID, and, if so, uses the InfoHubID associated with that global directory. If the current global directory path was never used in association with an InfoHub ID, the configuration fails with the IHHUBDESCREQ error.

Action: Always specify an InfoHub Descriptor in your configuration file. Otherwise, ensure that the current global directory was previously associated in an InfoHubID.

IHIDEVEN, An xLine ID must be odd

InfoHub Error: This error indicates that a configuration file attempted to use an even number for either a FileLine or PipeLine configuration.

Change the ID from an even number to an odd number. Even numbers are reserved for future development.

IHIDINVALID, An ID must be a non-zero positive GT.M canonical integer no more than 2**31-1

InfoHub Error: This error occurs when configuration processing detects an ID which is not a canonical GT.M integer, or an XLine process has an inappropriate even ID.

Action: Ensure that your descriptors contains IDs that are canonical GT.M integers, and that XLine IDs are odd except for PipeLines with a corresponding (one less) odd ID.

IHIDMISMATCH, A configuration file cannot modify a preexisting relationship between Name and ID

InfoHub Error: This error occurs when configuration processing detects an attempt to change a preexisting relationship between Name and ID.

Action: Specify the correct Name and ID and rerun configuration processing.

IHIDORNAMEREQ, A configuration cannot use something that does not have an explict or implicit reference to an InfoDictItem in an associated InfoDict

InfoHub Error: This error occurs when configuration processing detects there is no Name or ID for the InfoDict. If InfoDictItemID is specified, configuration processing verifies before configuration completion that the specified InfoDict domain maps the specified InfoDictItemName; This error also occurs when configuration processing detects there is no Name or ID for a InfoDictItem.

Action: Ensure your descriptors use an InfoDict Name or ID where required.

IHINFO, mmmm

InfoHub Info: The message indicates some information about the status of the service.

Action: The message mmmm is informational and may be associated with an error.

IHINVKEYFRMGLNR, Invalid key returned from dddd

InfoHub Fatal: A gleaner Expr returned an invalid key as described by dddd

Action: Report to the gleaner maintainer and provide the context information from error and the file that this error produces.

IHINVPROCCONF, Invalid process configuration: yyyy

InfoHub Error: xLines emit this error message when the configuration produces an incorrect value for yyyy.

Action: Consult the configuration file for correct values. Use the verbose and reload the configuration. If the configuration routine notes no flaws, contact support.

IHJOBENVRESTOREFAIL, Could not restore dddd to 'yyyy' errno=zzzz

InfoHub Fatal Message: The InfoHub JOB command framework was unable to restore the environment variable dddd to yyyy. The error code is zzzz. Because this error does not leave the InfoHub or Publisher service in an unintended state the service is shutdown.

Action: Please check the environment variable limit and increase it. If that does not work, consider eliminating unneeded environment variables.

IHJOBENVSETFAIL, Could not set dddd to 'yyyy' errno=zzzz

InfoHub Warning: The InfoHub JOB command framework was unable to set the environment variable dddd to yyyy. The error code is zzzz. This error is followed by an IHJOBFAIL from either a InfoHub or Publisher service.

Action: Please check the environment variable limit and increase it. If that does not work, consider eliminating unneeded environment variables.

IHJOBMSG, InfoHubJob framework message: dddd

InfoHub Information: The dddd portion of the message is describes some state of the InfoHub JOB framework

Action: None

IHJOBNORTN, Cannot JOB target label dddd because it does not exist

InfoHub Warning: The target label dddd does not exist. This should be followed by an IHJOBFAIL message from either the InfoHub or Publisher service.

Action: Please review the log files and contact support.

IHJOBSTARTDIR, Unable to create the JOB commands starting directory dddd; Error code yyyy

InfoHub Warning: The InfoHub JOB command framework was unable to create the target directory dddd. yyyy is the error returned from the underlying operating system. This should be followed by an IHJOBFAIL message from either the InfoHub or Publisher service.

Action: Please review the path to ensure that it has the correct permissions.

IHMETANOINFO, A META type InfoDictItem cannot have a NoInfo Subscription

InfoHub Error: This error occurs when configuration processing detects an item with "META" type with a corresponding NoInfo Subscription.

Action: META nodes act to aggregate multiple conditions into a single Subscription. Because META nodes use triggers and don't store data, and NoInfo Subscriptions use processes that periodically poll data, attempting to use a NoInfo Subscription on a META node produces an IHMETANOINFO error. NoInfo Subscriptions must refer to items with a type other than META or NONE.

IHMETANOSUB, A META type InfoDictItem must have a Subscription

InfoHub Error: This error occurs when configuration processing detects an item with "META" type but no corresponding Subscription.

Action: META nodes act to aggregate multiple conditions into a single Subscription. Ensure that you always specify a corresponding Subscription for a "META" type InfoDictItem.

IHMETANOTSUPRTD, META nodes not currently supported in PipeLines - all subsequent META nodes silently ignored

InfoHub Error: The current PipeLine does not support META type nodes.

Action: META type nodes provide a means to avoid large numbers of triggers by using gleaner logic to create a common trigger point for a class of events. If you have a use for this in a PipeLine, please contact FIS.

IHMISSINGCONF, Configuration missing for [PipeLine|FileLine]

InfoHub Fatal: The PipeLine of FileLine could not find its own configuration.

Action: Shutdown the InfoHub and verify that the configuration has the correct values, using the --action=extract. Reload the configuration and restart. If the configuration routine notes no flaws or the process exits with the same failure message, contact support.

IHMISSINGPARMS, Key startup parameters are missing for [InfoHubMain|Publisher|NoInfo]; The supplied parameter(s) dddd

InfoHub Error: An InfoHub service process was started without the appropriate parameters. The dddd portion of the message indicates the service process and passed parameters.

Action: If the service process was not started manually or programmatically, contact support. If started manually or programmatically please check the parameters.

IHMISSINGPROC, Process nnnn (ID:dddd PID:pppp) did not respond to $ZSIGPROC() from xxxx

InfoHub Warning: The service process nnnn with ID dddd and PID pppp did not respond to a signal from the parent service xxxx

Action: These errors typically indicate that an error occurred in the service process nnnn. Please refer to that process's log file for more diagnostic information and contact support. This message is also present if the process was killed.

IHMKTIMEFAIL, Could not convert time; external call to gtmposix.mktime() failed; path to gtmposix: pppp

InfoHub Error: This error indicates that the call to the GT.M POSIX plugin functionmktime (man mktime for more information) failed. pppp is the path to the External Call Table file.

Action: Please check the GT.M POSIX plugin configuration.

IHNAMEMISMATCH, This descriptor has an ID and Name that don't match another definition for the item

InfoHub Error: Configuration processing produces this error when the descriptor provides both the InfoHubID and the InfoHubName, and the ID is already in the configuration, but the specified Name does not match the existing Name.

Action:You cannot change the name of an existing InfoHub - you must either remove it, possibly with a Purge action, or create a new InfoHub with the desired name.

IHNAMEREQ, While configuration logic can generate IDs for Names, you must specify names except for InfoHubs and Subscribers

InfoHub Error: This error occurs when configuration processing cannot resolve an ID to an existing name. Descriptor syntax commonly provides fields for specifying both a name and an ID. When the descriptor contains only one of the two, the configuration process tries to resolve the other. If there is only a name, which does not currently exist, it generates an ID. This error occurs when there is only an ID (except for the InfoHub and Subscriber descriptors where it generates a random name) and InfoHub cannot resolve that ID to an existing name.

Action: Either specify both ID and name in the descriptor specification, or ensure that those entries that do not have a name can resolve based on other InfoDict and InfoDictItem descriptors.

IHNOACK, The InfoHub dddd (PID:pppp) did not acknowledge the request after ssss

InfoHub Error: The InfoHub dddd, process ID pppp, did not acknowledge a request from the command line. This message is printed after ssss seconds.

Action: Check the InfoHub's status. If the InfoHubMain process is not active contact support with information from the log file.

IHNOCHILD, The service ssss has no child configurations defined

InfoHub Warning: The service ssss was not configured with services to monitor.

Action: Check the configuration file to ensure that the service ssss has processes defined for it.

IHNOCOMPLETE, The InfoHub dddd (PID:pppp) did not complete the request after tttt

InfoHub Error: The InfoHub dddd, process ID pppp, did not acknowledge a request from the command line. This message is printed after tttt seconds.

Action: Check the InfoHub's status. If the InfoHubMain process is not active contact support with information from the log file.

IHNODUMMYGLD, Unable to create a global directory for environment descriptor testing: gggg

InfoHub Error: This is a configuration error message that indicates that the configuration routine was unable to create a temporary global directory while validating the configuration file.

Action: If the message gggg does not indicate the source of the problem, contact support.

IHNOERRSETUP, Error handler was not setup tttt

InfoHub Warning: A Subscription implemented with trigger tttt, but the xLine process that set off the trigger had not registered an error handler.

Action: Contact support for dealing with this issue.

IHNOGBLDIR, The value of $gtmgbldir or --gbldir 'gggg' does not point to a valid global directory

InfoHub Error: The InfoHub command line could not use gggg as a global directory.

Action: Check that the supplied path points to a valid global directory.

IHNOINFOINVPER, A NoInfo Condition must have a valid numeric Period in seconds

Configure Error: This error occurs when configuration processing detects a non-numeric Period for the "NoInfo" condition.

Action: Ensure that you specify Always specify a numeric value of seconds for Period in Subscription descriptor entry for a "NoInfo" condition .

IHNOINFONOVAL,A NoInfo Condition must have a Subscription

InfoHub Error: This error occurs when configuration processing encounters a Value for a "Noinfo" condition in the Subscription descriptor entry.

Action: Ensure that there is no Value specified for the "Noinfo" condition.

IHNORESTART, Service nnnn:ID=dddd,InfoHubID=iiii will not be restarted

InfoHub Warning: The InfoHub service named nnnn with ID dddd belonging to the InfoHub ID iiii will not be restarted due to five failures in the past 5 minutes.

Action: Any services that fail five time within five minutes typically indicates a configuration error. Check the full status listing of InfoHub services and the log files for information on why the service is failing.

IHNOSUBSCRIBER, There are no subscribers listed for ssss

InfoHub Warning: A Subscription ssss condition was met, but there are no Subscribers to notify.

Action: Check and remediate the configuration.

IHNOSUBSCRIPTION, There are no subscriptions defined for the NoInfo service ID dddd

InfoHub Warning: The NoInfo service has no subscriptions defined for it.

Action: Please check the configuration for the NoInfo subscription. If there are no NoInfo subscriptions, please contact support.

IHNOSUCHID, There is no Info Hub ID or name nnnn

InfoHub Error: The messages indicates that there is no InfoHub named nnnn or with the ID nnnn.

Action: Use the list command line action, --action=list, to list the names and numeric IDs of all configured InfoHubs.

IHNOTBINSH, The format of the information in the Env Descriptor must be acceptable to /bin/sh

InfoHub Error: InfoHub uses /bin/sh to establish its environments, but the ENV descriptor contained a command that /bin/sh rejected.

Action: Use /bin/sh to develop an appropriate command and modify the descriptor accordingly.

IHNOTCONFIGURED, The current global directory gggg does not have an active InfoHub configuration

InfoHub Error: The InfoHub command line detected that the global directory gggg does not have an active configuration

Action: Please verify that the target global directory has an InfoHub configured.

IHNOTINTRIG, Not inside a trigger

InfoHub Warning: InfoHub implements most Subscriptions with triggers, and supplies standard trigger logic to address many Subscription conditions. This error indicates that logic intended for use in a Subscription trigger was invoked by something other than a trigger.

Action: Consult custom plugin code for direct invocation of the routine InfoHubNotify. If none is found, please contact support

IHNOTRUNNING, The InfoHub nnnn is not running: tttt

InfoHub Error: The messages indicates that there is no InfoHub named nnnn is not running. The message tttt indicates the last time it was stopped.

Action: Start the InfoHub prior to executing commands for it.

IHNOXLINE4PUB, Publisher pppp has no xLine, .msg entry

InfoHub Error: The Publisher with ID pppp has no xLine processes and so cannot contribute useful information.

Action: Remove the Publisher if is no needed or appropriately configure a FileLine and/or PipeLine to generate information for this Publisher.

IHODDFLIDREQ, An even PipeLine descriptor must have a one-less odd PipeLine descriptor

InfoHub Error: Even PipeLine descriptors define a separate process to monitor the stderr of the process created for the PIPE device. Such descriptors must have the same PipeCMD and an even ID one higher than the descriptor for the main PipeLine process.

Action: Check your configuration and adjust as appropriate.

IHONEINFOHUBREQ, A configuration must have no more than one InfoHub Descriptor

InfoHub Error: This error occurs when configuration processing encounters more than one InfoHub Descriptor.

Ensure that you specify only one InfoHub Descriptor per configuration file.

IHORPHNDMETA, Orphaned meta update to nnnn

InfoHub Warning: This message indicates that an xLine was unable to file a META node.

Action: For META information to be filed, the data must be filed under a real node. Please check the InfoDict configuration hierarchy.

IHPARENTUNDEF, A Parent InfoDict must exist

InfoHub Error: A ParentInfoDictID indicates a Parent InfoDict domain contains an item definition for the Name of this InfoDict domain. This error occurs when configuration processing detects there is no definition for the specified Parent Name.

Action: Ensure that the parent Name has a InfoDict entry in the configuration file.

IHPARSERROR, An error occurred while parsing the comment line. Please refer to the following message for an explanation:

InfoHub Error: The InfoHub command line parser encountered an error.

Action: Please see the associated text and fix the parameter.

IHPATNOVAL, Because they have a patcode with the operator, pattern Conditions must not have a value

InfoHub Error: This error occurs when configuration processing detects a value with a pattern condition. The patcode following the pattern operator is not a value and so needs to be part of the condition.

Action: When using a pattern condition, place the patcode right after the pattern operator (?) in the condition field and leave the Value field empty.

IHPIPECMDMISMATCH, An even PipeLine descriptor must have the same Command as the one-less odd PipeLine descriptor

InfoHub Error: This error occurs when configuration processing detects an an even PipeLine descriptor having a different PipeCmd than its corresponding odd-numbered PipeLine descriptor.

Action: Ensure that you specify the same PipeCmd for every combination of odd and immediately subsequent even PipeLine descriptors.

IHPIPECMDREQ, PipeCMD is a required field for a PipeLine Descriptor

InfoHub Error: A PipeLine descriptor must include a command to invoke the PIPE device co-process.

Action: Add the appropriate command to the PipeLine descriptor.

IHPUBLISHERREQ, FileLine and PipeLine definitions require a Publisher

InfoHub Error: A FileLine or PipeLine descriptor must explicitly or implicitly refer to a Publisher descriptor by ID or Name.

Action: Review your configuration and adjust appropriately.

IHSEQNOMISMATCH, Sequence number for "SeqNoToTime" mapping is behind that of the actual value

InfoHub Warning: This warning indicates that the private copy of the sequence-number-to-time is behind the actual value stored in the database. This is an out-of-design condition for which the xLine takes corrective action to avoid failure.

Action: Inspect the database and custom plugin code for incorrect SETs/KILLs of InfoHubInfo nodes. Contact support if custom code is not the source of the problem.

IHSETENVFAIL, Could not setenv or unsetenv; external call to gtmposix.setenv() or gtmposix.unsetenv() failed; xxxx

InfoHub Fatal: A service process was unable to set or unset an environment variable.

Action: Please ensure that nothing has consumed the allotted environment variable memory space.

IHSIGNALVALFAIL, Could not obtain SIGUSR1; external call to gtmposix.signalval() failed

InfoHub Fatal: A InfoHub component could not get the platform specific value for SIGUSR1 from the gtmposix plugin.

Action: Please check to ensure that the gtmposix plugin was installed correctly.

IHSOMEEXPRREQ, FileLine and PipeLine descriptors must specify at least one of PreExpr, InfoExpr and PostExpr

InfoHub Error: This error occurs when configuration processing detects the FileLine descriptor does not specify at least one of PreExpr, InfoExpr and PostExpr.

Action: Always specify at least one of PreExpr, InfoExpr, or PostExpr in the FileLine descriptor.

IHSRVCLCKFAIL, Could not acquire the service lock

InfoHub Fatal: This message indicates that another service already holds the service lock.

Action: Use LKE to list all MUMPS process holding locks for the InfoHub and perform a full status listing. If no conflict can be found, contact support.

IHSUBCONDINV, A Subscription descriptor must specify a Condition using a GT.M binary operator, except concatenation, or the text 'NOINFO'

Configure Error: This error occurs when configuration processing encounters an invalid Condition in the Subscription descriptor entry.

Action: Ensure that Condition specifies a GT.M binary operator excluding concatenation and non-relational arithmetic operators, or the (case-insensitive) text: "Noinfo" which configures detection of a period during which the specified item receives no new data.

IHSUBINVPER, Period in a Subscription must be a numeric value in seconds

InfoHub Error: This error occurs when configuration processing encounters a non-numeric Period value for rate checking in the subscription descriptor entry.

Action:Always specify a numeric value of seconds for Period.

IHSUBSCRPTNNOPATH, A Subscription must have a valid path from a Publisher to the subscribed InfoDictItem

InfoHub Error: A Subscription must identify an InfoDictItem which in an InfoDict that has a path, possibly using parent relationships to a FileLine or PipeLine to a Publisher.

Action: Review your configuration and adjust appropriately.

IHSUBSPROB, IHSUBSPROB, Subscription must have a suitable comparison, but cccc isn't valid

InfoHub Error: The comparison value is not a valid expression or patcode

Action: Correct the Subscription Value

IHSUBVALREQLIT, Subscriptions except for pattern conditions and NoInfo must have a value to use with the Condition

InfoHub Error: This error occurs when configuration processing encounters an invalid Value in the Subscription descriptor entry.

Action: Ensure that Value specifies a GT.M literal which the Subscriber relates to the Item; it is required except for pattern match and "Noinfo" conditions.

IHTMPDBFAIL, Publisher pppp (SEQNO:ssss) was unable to setup the temporary directory

InfoHub Error: Publisher pppp, configuration sequence number ssss, could not create and populate the temporary directory.

Action: Check the path and the authorization for all elements in the path and creak or modify the authorizations of all of its directories. Alternatively change the configuration to use an appropriate path for the Publisher TempPWD.

IHTNONENOSUB, InfoDictItems with a type 'NONE' cannot support a Subscription

InfoHub Error: This error occurs when configuration processing detects a Subscription on an item with a type of "NONE".

Action: NONE items serve to construct paths where there is no data associated with some node on the path. Either avoid trying place a Subscription on a NONE type InfoDictItem, or configure the item to have data and an appropriate Type.

IHTRIGFAIL, Trigger tttt encountered an error, please see ffff for details

InfoHub Warning: Subscription trigger tttt failed and produced a context dump in file ffff.

Action: Examine the context file and analyze the failure characteristics, which might include schema dependencies and coding problems; consult your trigger maintainer as appropriate.

IHTRIGINSTALLFAIL, Installation of the trigger definition file ffff failed

InfoHub Fatal: An attempt to revise the Subscription triggers as defined in file ffff failed to load.

Action: Examine any accompanying messages or if necessary try the load with MUPIP TRIGGER or $ZTRIGGER() do get diagnostic messages. Consult your trigger maintainer as appropriate.

IHUNKNOWNERR, Supplied error mnemonic (mmmm) was not found

InfoHub Error: The error message used to generate an error is not known to the InfoHub. InfoHub error messages are listed in InfoHubErrorMessages.m. Plugin error messages are listed in IHPluginErrorMessage.m.

Action: Check custom or plugin code for the message mmmm. If not found, please contact support.

IHUNRESPPROC, Must re-signal pppp because it failed to respond in a timely manner

InfoHub Warning: Process with PID pppp seems to not acknowledge an action request.

Action: Check the process status, possibly with gtmpcat and take appropriate action. Note that xLine services spend a large amount of time processing data and may not respond in a timely fashion.

IHZEROIDINVALID, Zero (0) is not a valid ID

InfoHub Error: This error occurs when configuration processing detects an ID a value of ZERO (0).

Action: Ensure that the descriptor ID is a canonical positive integer from 1 to 2**31-1.

IHZLINKFAIL, Could not zlink rrrr, Please check the configured $ZROUTINES zzzz

InfoHub Error: The InfoHub was unable to zlink the file rrrr. It does not exist in the current $ZROutines zzzz.

Action: Verify that the routine rrrr exists in the $ZROutine paths.