Token (Lexer) - NetBeans API Javadoc (Current Development Version)

Overview

Package

Class

Use

Tree

Deprecated

Index

Help

org.netbeans.modules.lexer/2 1.19.0 1

PREV CLASS NEXT CLASS

FRAMES NO FRAMES

SUMMARY: NESTED | FIELD | CONSTR | METHOD

DETAIL: FIELD | CONSTR | METHOD

org.netbeans.api.lexer
Class Token<T extends TokenId>

java.lang.Object
  org.netbeans.api.lexer.Token<T>

public abstract class Token<T extends TokenId>
extends Object
extends Object

Token describes a lexical element of input text.
It mainly provides an identification by id() and a textual body (aka token's image) by text().
Only lexers should produce token instances and they should do it solely by using methods of TokenFactory.

Note: Do not create custom extensions of this class - lexers may only return implementations produced by TokenFactory. Creation of any other token implementations will be refused.

Token guarantees stability of the id() and length() methods. The hashCode() and equals(Object) methods use the default implementations from java.lang.Object.
The two tokens are only equal if they are the same object.

Constructor Summary

protected Token()
          Create token instance.

Method Summary

boolean equals(Object o)
          Make sure the default implementation of equals() is used and the token can safely be used in maps.

abstract Object getProperty(Object key)
          Get extra property of this token.

int hashCode()
          Make sure the default implementation of hashCode() is used and the token can safely be used in maps.

abstract boolean hasProperties()
          Quickly determine whether this token has any extra properties.

abstract T id()
          Get identification of this token.

abstract boolean isCustomText()
          Check whether text() returns a custom value that may differ from the original content of the text input.

abstract boolean isFlyweight()
          Checks whether this token instance is used for multiple occurrences of this token in this or other inputs.

abstract boolean isPreprocessedText()
          Check whether this token has preprocessed text (e.g.

abstract int length()
          Get number of characters in the original text input that the token spans.

abstract int offset(TokenHierarchy<?> tokenHierarchy)
          Get the offset at which this token is present in the input or -1 if this token is flyweight (and therefore does not store offset).

abstract PartType partType()
          Check whether this token represents a complete token or whether it's a part of a complete token.

abstract CharSequence preprocessedText()
          Get a text of this token as it was preprocessed by a character preprocessor (e.g.

abstract String preprocessError()
          Get an description of the error that occurred during preprocessing of the token's characters.

abstract int preprocessErrorIndex()
          Get the index relative to the token's begining (in the original input text) of where the preprocessor error has occurred.

abstract CharSequence text()
          Get text of this token (aka token's image) as a character sequence.

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, toString, wait, wait, wait

Constructor Summary
`protected`	`Token()` Create token instance.

Method Summary
`boolean`	`equals(Object o)` Make sure the default implementation of `equals()` is used and the token can safely be used in maps.
`abstract Object`	`getProperty(Object key)` Get extra property of this token.
`int`	`hashCode()` Make sure the default implementation of `hashCode()` is used and the token can safely be used in maps.
`abstract boolean`	`hasProperties()` Quickly determine whether this token has any extra properties.
`abstract T`	`id()` Get identification of this token.
`abstract boolean`	`isCustomText()` Check whether `text()` returns a custom value that may differ from the original content of the text input.
`abstract boolean`	`isFlyweight()` Checks whether this token instance is used for multiple occurrences of this token in this or other inputs.
`abstract boolean`	`isPreprocessedText()` Check whether this token has preprocessed text (e.g.
`abstract int`	`length()` Get number of characters in the original text input that the token spans.
`abstract int`	`offset(TokenHierarchy<?> tokenHierarchy)` Get the offset at which this token is present in the input or `-1` if this token is flyweight (and therefore does not store offset).
`abstract PartType`	`partType()` Check whether this token represents a complete token or whether it's a part of a complete token.
`abstract CharSequence`	`preprocessedText()` Get a text of this token as it was preprocessed by a character preprocessor (e.g.
`abstract String`	`preprocessError()` Get an description of the error that occurred during preprocessing of the token's characters.
`abstract int`	`preprocessErrorIndex()` Get the index relative to the token's begining (in the original input text) of where the preprocessor error has occurred.
`abstract CharSequence`	`text()` Get text of this token (aka token's image) as a character sequence.

Methods inherited from class java.lang.Object
`clone, finalize, getClass, notify, notifyAll, toString, wait, wait, wait`

Constructor Detail

Token

protected Token()

Create token instance.

Throws:: IllegalStateException - if a non-lexer-module-implementation token is attempted to be created.

Method Detail

id

public abstract T id()

Get identification of this token.

Returns:: non-null identification of this token.

text

public abstract CharSequence text()

Get text of this token (aka token's image) as a character sequence.
This text usually corresponds to the characters present in the lexed text input unless isCustomText() returns true.

Note for mutable input sources:
This method should only be called within a readonly (or read-write) transaction over the underlying input source (such as javax.swing.text.Document.render() for Swing documents).
The result returned by this method is only valid within a readonly (or read-write) transaction over the input source (method must be re-called during the next readonly transaction).

Returns:

non-null, non-empty text of this token. It may be null in case the token was used for a mutable input and it was removed from the token list for the given input (but even in such case the text can be retained in certain cases).

The behavior of equals() and hashCode() of the returned character sequence is generally undefined.
The returned character sequence can NOT be compared to another character sequence by using its equals() method.
TokenUtilities contains utility methods related to token text comparing.

The returned text is just a pointer to the primary source of the data e.g. a swing document. The character data are not duplicated in the tokens.

See Also:

preprocessedText()

isCustomText

public abstract boolean isCustomText()

Check whether text() returns a custom value that may differ from the original content of the text input.
Using custom text may be useful in case when only certain part of the token is useful for the particular use and the token's text can be shrinked and possibly a flyweight text can be used.
Also this is useful when using lexers generated by various lexer generators that generally allow to use a custom text in the produced tokens.

Returns:: true if the text of the token does not correspond to the original characters present in the text input being lexed.

length

public abstract int length()

Get number of characters in the original text input that the token spans.
Usually this is the same value like text().length() unless isCustomText() returns true.
Also this method will return valid length in all cases even when the text of the token could become null.

Returns:: >=0 length of the token.

offset

public abstract int offset(TokenHierarchy<?> tokenHierarchy)

Get the offset at which this token is present in the input or -1 if this token is flyweight (and therefore does not store offset).
Note: Use of TokenSequence.offset() is usually preferred over this method.
For flyweight tokens the real offset of the token may only be determined by doing TokenSequence.offset() when positioned on the particular flyweight token.
If necessary the flyweight token may be replaced by regular token by using TokenSequence.offsetToken().

The complexity of the method should generally be constant regardless of the level of the language embedding.

Parameters:: tokenHierarchy - token hierarchy to which the offset computation will be related. It may either be the live token hierarchy (which is equivalent to passing null) or a snapshot of the original token hierarchy. For other values the result is generally undefined.
Returns:: >=0 offset of the token in the input or -1 if this token is flyweight.

isFlyweight

public abstract boolean isFlyweight()

Checks whether this token instance is used for multiple occurrences of this token in this or other inputs.
For example keywords or operators are typically flyweight tokens while e.g. identifiers are not flyweight as their text generally varies.
Flyweight tokens may decrease the memory consumption for the tokens considerably for frequently used tokens. For example a single space ' ' may be a useful flyweight token as it's used very often throughout a source. The decision of what tokens are made flyweight is upon the implementor of the particular language.

If the token is flyweight its offset(TokenHierarchy) returns -1.

Returns:: true if the token is flyweight or false otherwise.

partType

public abstract PartType partType()

Check whether this token represents a complete token or whether it's a part of a complete token.

isPreprocessedText

public abstract boolean isPreprocessedText()

Check whether this token has preprocessed text (e.g. Unicode escapes in the token's text were translated).

Returns:: true if this token contains preprocessed text and its preprocessedText() returns a valid result.

preprocessedText

public abstract CharSequence preprocessedText()

Get a text of this token as it was preprocessed by a character preprocessor (e.g. with Unicode escapes translated).

preprocessError

public abstract String preprocessError()

Get an description of the error that occurred during preprocessing of the token's characters.

preprocessErrorIndex

public abstract int preprocessErrorIndex()

Get the index relative to the token's begining (in the original input text) of where the preprocessor error has occurred.

Returns:: >=0 index where the character preprocessing error has occurred. Returns -1 if there was no preprocessing error.

hasProperties

public abstract boolean hasProperties()

Quickly determine whether this token has any extra properties.

getProperty

public abstract Object getProperty(Object key)

Get extra property of this token.
The token properties are defined by the lexer upon token creation. The clients of the API cannot set any property of the token.

Parameters:: key - non-null key of the property to get.
Returns:: non-null value of the property or null if the property does not have any value.
See Also:: hasProperties()

hashCode

public final int hashCode()

Make sure the default implementation of hashCode() is used and the token can safely be used in maps.

Overrides:: hashCode in class Object

equals

public final boolean equals(Object o)

Make sure the default implementation of equals() is used and the token can safely be used in maps.

Overrides:: equals in class Object