CharPreprocessor (Lexer) - NetBeans API Javadoc (Current Development Version)

Overview

Package

Class

Use

Tree

Deprecated

Index

Help

org.netbeans.modules.lexer/2 1.19.0 1

PREV CLASS NEXT CLASS

FRAMES NO FRAMES

SUMMARY: NESTED | FIELD | CONSTR | METHOD

DETAIL: FIELD | CONSTR | METHOD

org.netbeans.spi.lexer
Class CharPreprocessor

java.lang.Object
  org.netbeans.spi.lexer.CharPreprocessor

public abstract class CharPreprocessor
extends Object
extends Object

Character preprocessor allows to translate a sequence of characters to a single character so it may be used for example for Unicode sequences translation.
If there are any preprocessed characters for a particular token then a special token instance get created that provides the preprocessed chars.

Constructor Summary

CharPreprocessor()


Method Summary

static CharPreprocessor createUnicodeEscapesPreprocessor()
          Create instance of character preprocessor for Unicode escape sequences.

protected void inputBackup(int count)
          Backup a given number of input characters.

protected int inputRead()
          Read a single character for preprocessing from the underlying input.

protected abstract boolean isSensitiveChar(char ch)
          Check whether the given character may be part of the sequences preprocessed by this preprocessor.

protected abstract int maxLookahead()
          Return maximum number of extra characters (not being part of the recognized sequence) that this preprocessor may look ahead in order to recognize the preprocessed character sequence.

protected void notifyError(String errorMessage)
          Notify the error that occurred during preprocessing of the current character.

protected void outputOriginal(int ch)
          Output the character as it was read from the input.

protected void outputPreprocessed(char ch, int extraInputLength)
          Output preprocessed character.

protected abstract void preprocessChar()
          Preprocess at least one character of the input.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Summary
`CharPreprocessor()`

Method Summary
`static CharPreprocessor`	`createUnicodeEscapesPreprocessor()` Create instance of character preprocessor for Unicode escape sequences.
`protected void`	`inputBackup(int count)` Backup a given number of input characters.
`protected int`	`inputRead()` Read a single character for preprocessing from the underlying input.
`protected abstract boolean`	`isSensitiveChar(char ch)` Check whether the given character may be part of the sequences preprocessed by this preprocessor.
`protected abstract int`	`maxLookahead()` Return maximum number of extra characters (not being part of the recognized sequence) that this preprocessor may look ahead in order to recognize the preprocessed character sequence.
`protected void`	`notifyError(String errorMessage)` Notify the error that occurred during preprocessing of the current character.
`protected void`	`outputOriginal(int ch)` Output the character as it was read from the input.
`protected void`	`outputPreprocessed(char ch, int extraInputLength)` Output preprocessed character.
`protected abstract void`	`preprocessChar()` Preprocess at least one character of the input.

Methods inherited from class java.lang.Object
`clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

Constructor Detail

CharPreprocessor

public CharPreprocessor()

Method Detail

createUnicodeEscapesPreprocessor

public static CharPreprocessor createUnicodeEscapesPreprocessor()

Create instance of character preprocessor for Unicode escape sequences.

preprocessChar

protected abstract void preprocessChar()

Preprocess at least one character of the input.
Preprocessor must always preprocess at least one input character per invocation but only a minimum necessary number of characters should be preprocessed by each invocation of this method.

Example:

   public void preprocessChar() {
     switch (ch = inputRead()) {
       case '\\': // possible start of sequence
         switch (ch = inputRead()) {
           case 'u': // start of escape sequence
             ... // read the whole sequence
             outputPreprocessed(prepCh, extraInputLength);
             break;
           default:
             outputOriginal('\\');
             outputOriginal(ch);
             break;
         }
         break;
       default:
         outputOriginal(ch);
     }
   }

The processor is only designed to do several-chars-to-one translation.
It is not designed to return more than one character for a single input char.
Also if the character is really preprocessed it must be composed from at least two input characters (see extraInputLength parameter of outputPreprocessed(char, int).

The preprocessor must be able to process all the characters given to it on input. However it should not preprocess EOF in any way - the EOF is just information that there is an end of the input and any possibly unfinished escape sequence needs to be translated in a reasonable way.
Once all the characters prior EOF were preprocessed the EOF should be returned by outputOriginal(int).

isSensitiveChar

protected abstract boolean isSensitiveChar(char ch)

Check whether the given character may be part of the sequences preprocessed by this preprocessor.
The infrastructure may use this method to test whether it can start relexing starting at a particular position.

maxLookahead

protected abstract int maxLookahead()

Return maximum number of extra characters (not being part of the recognized sequence) that this preprocessor may look ahead in order to recognize the preprocessed character sequence.
For example for unicode escape sequences the returned number is 1 (see UnicodeEscapesPreprocessor implementation for details).

inputRead

protected final int inputRead()

Read a single character for preprocessing from the underlying input.

Returns:: valid character or LexerInput.EOF if there are no more characters available on the input.

inputBackup

protected final void inputBackup(int count)

Backup a given number of input characters.

Parameters:: count - >=0 number of chars to backup.

outputOriginal

protected final void outputOriginal(int ch)

Output the character as it was read from the input.
By using this method the infrastructure knows that the character is the same like the original character read by inputRead().

outputPreprocessed

protected final void outputPreprocessed(char ch,
                                        int extraInputLength)

Output preprocessed character. There is usually more than one input character forming a single preprocessed character.

Parameters:: ch - preprocessed character.; extraInputLength - >0 number of extra input characters (besides a single character) that form the preprocessed character.
For example for unicode escape sequence " " it's 6-1=5.
The number is expected to be greater than zero (otherwise the present implementation would not work correctly). which should be fine for the known implementations (if not please request an API change).

notifyError

protected final void notifyError(String errorMessage)

Notify the error that occurred during preprocessing of the current character.

Parameters:: errorMessage - non-null error description.