PObserve: Log Parser

What is a PObserve Log Parser?

In order to use PObserve, a log parser is an essential component that converts system-generated log lines to PObserve events which are later consumed by a PObserve monitor. The log parser takes a log line (text or JSON) as input and converts it into a PObserveEvent object.

Input: A Log Line

• PObserve parser accepts a log line and converts each log line to an object. The log line can be in text or JSON format.

Output: Stream of PObserve Events

• Once the parser processes a log line, it must create a PObserve event from the extracted data and return it as a stream of events.

PObserve Event Class Structure

PObserve Event (Please check the Java Class here)

Field	Description
key	The "key" that will be used to partition events
timestamp	Timestamp of the event
event	P event object
logLine	log line that generates this PObserve event
customPayload	Additional payload that can be attached to this event

Writing a Log Parser

This section explains how to implement a parser for a lock server example (complete implementation of the parser here).

[Step 1] Create a parser class

1. Create a new public class that implements the parser interface
2. Override the apply method of the Parser interface

Skeleton code to implement a parser:

import com.amazon.pobserve.commons.Parser;

public class LockServerParser implements Parser<PEvent> {
   @Override
    public Stream<PObserveEvent<PEvent>> apply(Object logLine) {
        /* Implement the logic to convert log lines to events */
    }
}

[Step 2] Parse log line to PObserve Event

Important: PEvent Java Classes Generation

The PEvent java classes used in your parser are generated by compiling your written P specifications in pobserve mode (Run p compile --mode pobserve in your P package). You must first compile your P specifications in PObserve mode and then use the definitions in the PEvents.java file of the generated code. Create PEvent instances by filling them with the corresponding data extracted from the log line.

1. Inside the apply function, implement the logic that converts the logLine object to a PObserve Event
2. Extract all the required data from the log line and create P Events from them using the generated PEvent classes
3. Create PObserve events from the P event and return it as a stream

Warning

Since PObserve uses multi-threading for performance, do NOT use static variables in your parser as they can cause thread safety issues and unpredictable behavior.

Here is an example of parser code that parses a Lock Request from the lock server logs to a stream of PObserve Events:

import com.amazon.pobserve.commons.Parser;
import com.amazon.pobserve.commons.PObserveEvent;

public class LockServerParser implements Parser<PEvent> {

   private PEvents.eLockReq onLockReq(long clientID, long lockID, long lockID) {
        return new PEvents.eLockReq(new PTypes.PTuple_clnt_lock_rId(clientID, lockID, rId));
   }

   @Override
    public Stream<PObserveEvent<PEvent>> apply(Object logLine) {
        /* Implement the logic to extract type, clientID, lockID, lockID from the log line */
        PEvent event = null;
        if (type is a lock request) {
            PEvent event = onLockRes(clientID, lockID, rId, result, lockStatus);
        }
        /* Implement the logic to create Events of other types */
        String key = Long.toString(lockID);
        return event == null ? Stream.empty() : Stream.of(new PObserveEvent(key, time, event, line));
    }
}

Note

The key used while creating the PObserve event will be used for partitioning the events to appropriate monitors.

Refer to the completed LockServerParser in the LockServerPObserve package.