Journal Extract Formats

Journal EXTRACT files always start with a label to identify the format of the file. Simple and detail EXTRACT files have different labels, and the labels are specific to a GT.M release.

If the environment variable gtm_chset is set of UTF-8, then file format label is followed by another label called “UTF-8” to indicate UTF-8 mode.

After this label, the journal record extracts follow. These journal record extracts include fields or pieces delimited by a back slash ().

The first piece of an -EXTRACT output record contains a two-digit decimal transaction record type (for example, 01 for a process initialization record). The second piece contains the full date and time of the operation, represented in the $HOROLOG format. The third piece contains a GT.M assigned number (database transaction number) which uniquely identifies the transaction within the time covered by the journal file. The fourth piece contains the process ID (PID) of the process that performed the operation, represented as a decimal number. The remainder of the record depends on the record type.

Records of type SET, KILL, ZKILL, TSTART, and TCOMMIT include the token_seq as part of the output. It is the sixth field in the output of the journal record extract. When replication is in use, token_seq is a journal sequence number (jsnum) that uniquely identifies each transaction(for more information on journal sequence number refer to Chapter7: “Database Replication”). When replication is not in use and the transaction is a TP transaction, token_seq is an 8-byte token that uniquely identifies the entire TP transaction. For non-replicated, non-TP journal records, token_seq has a zero (0) value.

The format of the plain journal extract is as follows:

NULL    00\time\tnum\pid\clntpid\jsnum\strm_num\strm_seq
PINI(U) 01\time\tnum\pid\nnam\unam\term\clntpid\clntnnam\clntunam\clntterm
PINI(V) 01\time\tnum\pid\nnam\unam\term\mode\logintime\image_count\pname\clntpid\clntnnam\clntunam\clntterm\clntmode\clntlogintime\clntimage_count\clntpname
PFIN    02\time\tnum\pid\clntpid
EOF     03\time\tnum\pid\clntpid\jsnum
KILL    04\time\tnum\pid\clntpid\token_seq\strm_num\strm_seq\updnum\nodeflags\node
SET     05\time\tnum\pid\clntpid\token_seq\strm_num\strm_seq\updnum\nodeflags\node=sarg
ZTSTART 06\time\tnum\pid\clntpid\token
ZTCOM   07\time\tnum\pid\clntpid\token\partners
TSTART  08\time\tnum\pid\clntpid\token_seq\strm_num\strm_seq
TCOM    09\time\tnum\pid\clntpid\token_seq\strm_num\strm_seq\partners\tid
ZKILL   10\time\tnum\pid\clntpid\token_seq\strm_num\strm_seq\updnum\nodeflags\node
ZTWORM  11\time\tnum\pid\clntpid\token_seq\strm_num\strm_seq\updnum\ztwormhole
ZTRIG   12\time\tnum\pid\clntpid\token_seq\strm_num\strm_seq\updnum\nodeflags\node
LGTRIG  13\time\tnum\pid\clntpid\token_seq\strm_num\strm_seq\updnum\trigdefinition

where:

01 record indicates a process/image-initiated update (PINI) into the current journal file for the first time.

02 record indicates a process/image dropped interest (PFIN) in the current journal file.

03 record indicates all GT.M images dropped interest in this journal file and the journal file was closed normally.

04 record indicates a database update caused by a KILL command.

05 record indicates a database update caused by a SET command.

06 record indicates a ZTSTART command.

07 record indicates a ZTCOMMIT command.

08 record indicates a TSTART command.

09 record indicates a TCOMMIT command.

10 record indicates a database update caused by a ZKILL command.

11 records indicates a value for/from $ZTWORMHOLE (when replication is turned on).

12 record indicates a ZTRIGGER command.

13 record indicates a trigger definition as a logical action from an originating/primary instance to a replicating/secondary instance

The listing below shows an example of a simple journal extract:

01\67628,22261\3\1474136\gtmhost\gtmuser\0\0\\\
05\67628,22261\3\1474136\0\0\0\0\0\0\^global="newvalue"
02\67628,22261\4\1474136\0
03\67628,22261\4\1474136\0\0

Journal extracts contain NULL records only in a multisite replication configuration where triggers or external M-filters are active. Here are two examples when NULL records are sent to the journal files:

An external filter on an instance transforms a SET record to a NULL record that has a different schema.
If the source side has triggers enabled and its receiver side either runs a pre-trigger version of GT.M or runs on a platform where triggers are not supported, trigger definition journal records from the source side are transformed to NULL records on the receiver side.


[Important]	Important
A NULL record does not have global information. Therefore, it resides in the alphabetically last replicated region of the global directory.

The format of the detail journal extract is as follows:

PINI(U)  time\tnum\chksum\pid\nnam\unam\term\clntpid\clntnnam\clntunam\clntterm
PINI(V)  time\tnum\chksum\pid\nnam\unam\term\mode\logintime\image_count\pname\clntpid\clntnnam\clntunam\clntterm\clntmode\clntlogintime\clntimage_count\clntpname
PFIN     time\tnum\chksum\pid\clntpid
EOF      time\tnum\chksum\pid\clntpid\jsnum
SET      time\tnum\chksum\pid\clntpid\token_seq\strm_num\strm_seq\updnum\nodeflags\node=sarg
KILL     time\tnum\chksum\pid\clntpid\token_seq\strm_num\strm_seq\updnum\nodeflags\node
ZKILL    time\tnum\chksum\pid\clntpid\token_seq\strm_num\strm_seq\updnum\nodeflags\node
ZTWORM   time\tnum\chksum\pid\clntpid\token_seq\strm_num\strm_seq\updnum\ztwormhole
ZTRIG    time\tnum\chksum\pid\clntpid\token_seq\strm_num\strm_seq\updnum\nodeflags\node
TSTART   time\tnum\chksum\pid\clntpid\token_seq\strm_num\strm_seq
TSET     time\tnum\chksum\pid\clntpid\token_seq\strm_num\strm_seq\updnum\nodeflags\node=sarg
TKILL    time\tnum\chksum\pid\clntpid\token_seq\strm_num\strm_seq\updnum\nodeflags\node
TZKILL   time\tnum\chksum\pid\clntpid\token_seq\strm_num\strm_seq\updnum\nodeflags\node
TZTWORM  time\tnum\chksum\pid\clntpid\token_seq\strm_num\strm_seq\updnum\ztwormhole
TZTRIG   time\tnum\chksum\pid\clntpid\token_seq\strm_num\strm_seq\updnum\nodeflags\node
TLGTRIG  time\tnum\chksum\pid\clntpid\token_seq\strm_num\strm_seq\updnum\trigdefinition
USET     time\tnum\chksum\pid\clntpid\token_seq\strm_num\strm_seq\updnum\nodeflags\node=sarg
UKILL    time\tnum\chksum\pid\clntpid\token_seq\strm_num\strm_seq\updnum\nodeflags\node
UZKILL   time\tnum\chksum\pid\clntpid\token_seq\strm_num\strm_seq\updnum\nodeflags\node
UZTWORM  time\tnum\chksum\pid\clntpid\token_seq\strm_num\strm_seq\updnum\ztwormhole
UZTRIG   time\tnum\chksum\pid\clntpid\token_seq\strm_num\strm_seq\updnum\nodeflags\node
ULGTRIG  time\tnum\chksum\pid\clntpid\token_seq\strm_num\strm_seq\updnum\trigdefinition
TCOM     time\tnum\chksum\pid\clntpid\token_seq\strm_num\strm_seq\partners\tid
INCTN    time\tnum\chksum\pid\clntpid\opcode\incdetail
EPOCH    time\tnum\chksum\pid\clntpid\jsnum\blks_to_upgrd\free_blocks\total_blks\offset\fully_upgraded[\strm_num\strm_seq]...
PBLK     time\tnum\chksum\pid\clntpid\blknum\bsiz\blkhdrtn\ondskbver
AIMG     time\tnum\chksum\pid\clntpid\blknum\bsiz\blkhdrtn\ondskbver
NULL     time\tnum\chksum\pid\clntpid\jsnum\strm_num\strm_seq
ZTSTART  time\tnum\chksum\pid\clntpid\token
FSET     time\tnum\chksum\pid\clntpid\token_seq\strm_num\strm_seq\updnum\nodeflags\node=sarg
FKILL    time\tnum\chksum\pid\clntpid\token_seq\strm_num\strm_seq\updnum\nodeflags\node
FZKILL   time\tnum\chksum\pid\clntpid\token_seq\strm_num\strm_seq\updnum\nodeflags\node
GSET     time\tnum\chksum\pid\clntpid\token_seq\strm_num\strm_seq\updnum\nodeflags\node=sarg
GKILL    time\tnum\chksum\pid\clntpid\token_seq\strm_num\strm_seq\updnum\nodeflags\node
GZKILL   time\tnum\chksum\pid\clntpid\token_seq\strm_num\strm_seq\updnum\nodeflags\node
ZTCOM    time\tnum\chksum\pid\clntpid\token\partners
ALIGN    time\tnum\chksum\pid\clntpid

where:

AIMG records are unique to DSE action and exist because those actions do not have a “logical” representation.

EPOCH records are status records that record information related to check pointing of the journal.

NCTN records are the transaction numbers of the sequence of critical sections in which the process and marked the database blocks of the globals as previously used but no longer in use in the bit maps.

PBLK records are the before image records of the bit maps.

ALIGN records pad journal records so every alignsize boundary (set with MUPIP SET -JOURNAL and is visible in DSE DUMP -FILEHEADER output) in the journal file starts with a fresh journal record. The sole purpose of these records is to help speed up journal recovery.

Legend (All hexadecimal fields have a 0x prefix. All numeric fields otherwise are decimal):


`tnum`	Transaction number
`chksum`	Checksum for the record.
`fully_upgraded`	1 if the db was fully upgraded (indicated by a dse dump -file -all) at the time of writing the EPOCH
`pid`	Process id that wrote the jnl record.
`clntpid`	If non-zero, clntpid is the process id of the GT.CM client that initiated this update on the server side.
`jsnum`	Journal sequence number.
`token`	Unique 8-byte token.
`strm_num`	If replication is true and this update originated in a non-supplementary instance but was replicated to and updated a supplementary instance, this number is a non-zero value anywhere from 1 to 15 (both inclusive) indicating the non-supplementary stream number. In all other cases, this stream # value is 0. In case of an EPOCH record, anywhere from 0 to 16 such “strm_num” numbers might be displayed depending on how many sources of supplementary instance replication have replicated to the instance in its lifetime.
`strm_seq`	If replication is true and this update originated in a non-supplementary instance but was replicated to and updated a supplementary instance, this is the journal sequence number of the update on the originating non-supplementary instance. If replication is true and this update originated in a supplementary instance, this is the journal sequence number of the update on the originating supplementary instance. In all other cases, this stream sequence number is 0. Note that the journal seqno is actually 1 more than the most recent update originating on that stream number. In case of an EPOCH record, anywhere from 0 to 16 such “strm_seq” numbers might be displayed depending on how many sources of supplementary instance replication have replicated to the instance in its lifetime.
`tid`	TRANSACTIONID string (BATCH or any string of descriptive text chosen by the application) specified as an argument of the corresponding TSTART command. If TRANSACTIONID is not specified with TSTART, GT.M sets `tid` to null. TRANSACTIONID can specify any value for `tid` but affects GT.M behavior only when TRANSACTIONID specifies BATCH or BA.
`token_seq`	If replication is turned on, it is the journal sequence number. If not, it is a unique 8-byte token.
`trigdefinition`	Trigger definition string corresponding to an LGTRIG journal record.
`updnum`	=n where this is the nth update in the TP transaction. n=1 for the 1st update etc. 0 for non-TP.
`nodeflags`	Decimal number interpreted as a binary mask.. Currently only 5 bits are used. * 00001 (1) => update journaled but NOT replicated (For example, update inside a trigger) * 00010 (2) => update to a global that had at least one trigger defined, even if no trigger matched this update * 00100 (4) => $ZTWORMHOLE holds the empty string ("") at the time of this update or was not referenced during this update * 01000 (8) => update did not invoke any triggers even if they existed (For example, MUPIP LOAD) * 10000 (16) => whether the update (set or kill) is a duplicate. In case of a KILL, it is a kill of some non-existing node aka duplicate kill. Note that the dupkill occurs only in case of the Update Process. In case of GT.M, the KILL is entirely skipped. In both cases (duplicate set or kill), only a jnl record is written, the db is untouched. Combinations of the above bits would mean each of the individual bit characteristics. For example, 00011 => update within a trigger context, and to a global with at least one trigger defined. Certain bit combinations are impossible. For example, 01001 since GT.M replicates any update that does not invoke triggers.
`node`	Key that is being updated in a SET or KILL.
`sarg`	Right-hand side argument to the SET (that is, the value that the key is being SET to).
`partners`	Number of journaled regions participating in this TP (TCOM/ZTCOM record written in this TP) .
`opcode`	Inctn opcode. See gdsfhead.h inctn_opcode_t for all possible values.
`blknum`	Block number corresponding to a PBLK or AIMG or INCTN record.
`bsiz`	Block size from the header field of a PBLK or AIMG record.
`blkhdrtn`	Transaction number from the block header of a PBLK or AIMG record.
`ondskbver`	On disk block version of this block at the time of writing the PBLK or AIMG record.
`incdetail`	0 if opcode=1,2,3; blks2upgrd if opcode=4,5,6; blknum if opcode=7,8,9,10,11,12,13
`ztwormhole`	string corresponding to $ZTWORMHOLE
`blks2upgrd`	# of V6 format index blocks csd->blks_to_upgrd if opcode=6
`uname`	Name of the user that wrote this PINI record.
`clntunam`	If non-empty, clntunam is the name of the GT.CM client that initiated this update on the server side.
`offset`	offset correction to convert v6->v6p blocks in upgrade interregnum (at time of writing the EPOCH).

After the header(s), each line of a detail journal extract file begins with two hexadecimal numbers. The first of these is the offset in the journal file of the record being printed, and the second (enclosed in square brackets) is the length of that record. The listing below shows an example of a detail extract:

0x000106b8 [0x00d0] :: PINI   \67628,22261\3\4046488676\1474136\gtmhost\gtmuser\0\0\\\
0x00010788 [0x0050] :: PBLK   \67628,22261\3\4259595182\1474136\0\3\33\1\4
0x000107d8 [0x0050] :: SET    \67628,22261\3\4138852170\1474136\0\0\0\0\0\0\^global="value"
0x00010828 [0x00c8] :: EPOCH  \67628,22261\4\1321506953\1474136\0\0\0\96\101\0\1
0x000108f0 [0x0020] :: PFIN   \67628,22261\4\4268027422\1474136\0
0x00010910 [0x0028] :: EOF    \67628,22261\4\3238298114\1474136\0\0