TokenHierarchy (Lexer) - NetBeans API Javadoc (Current Development Version)

Overview

Package

Class

Use

Tree

Deprecated

Index

Help

org.netbeans.modules.lexer/2 1.19.0 1

PREV CLASS NEXT CLASS

FRAMES NO FRAMES

SUMMARY: NESTED | FIELD | CONSTR | METHOD

DETAIL: FIELD | CONSTR | METHOD

org.netbeans.api.lexer
Class TokenHierarchy<I>

java.lang.Object
  org.netbeans.api.lexer.TokenHierarchy<I>

public final class TokenHierarchy<I>
extends Object
extends Object

Token hierarchy represents a given input source as a browsable hierarchy of tokens.
It's is an entry point into the Lexer API.
It allows to create token sequences for hierarchy exploration and watching for token changes by attaching the token hierarchy listeners.
The hierarchy may either be flat or it can be a tree if the corresponding language hierarchy contains language embeddings.

Token hierarchy may also act as a snapshot of another "live" token hierarchy.
The snapshot may be created at any time by using createSnapshot() on the live token hierarchy.

Method Summary

void addTokenHierarchyListener(TokenHierarchyListener listener)
          Add listener for token changes inside this hierarchy.

static <T extends TokenId> TokenHierarchy<Void> create(CharSequence inputText, boolean copyInputText, Language<T> language, Set<T> skipTokenIds, InputAttributes inputAttributes)
          Create token hierarchy for the given input text.

static TokenHierarchy<Void> create(CharSequence inputText, Language<? extends TokenId> language)
          Create token hierarchy for the given non-mutating input text (for example java.lang.String).

static <T extends TokenId> TokenHierarchy<Void> create(Reader inputReader, Language<T> language, Set<T> skipTokenIds, InputAttributes inputAttributes)
          Create token hierarchy for the given reader.

TokenHierarchy<I> createSnapshot()
          Create a snapshot of the present mutable token hierarchy.

static <D extends Document> TokenHierarchy<D> get(D doc)
          Get or create mutable token hierarchy for the given swing document.

boolean isActive()
          Token hierarchy may be set inactive to release resources consumed by tokens.

boolean isMutable()
          Whether input text of this token hierarchy is mutable or not.

boolean isSnapshot()
          Check whether this token hierarchy is a snapshot.

boolean isSnapshotReleased()
          Check whether this snapshot is released.

Set<LanguagePath> languagePaths()
          Get a set of language paths used by this token hierarchy.

I mutableInputSource()
          Get mutable input source providing text over which this token hierarchy was constructed.

void removeTokenHierarchyListener(TokenHierarchyListener listener)
          Remove listener for token changes inside this hierarchy.

TokenHierarchy<I> snapshotOf()
          If this token hierarchy is snapshot then return the token hierarchy for which this snapshot was constructed.

void snapshotRelease()
          Release snapshot - should only be called if this token hierarchy is a snapshot.

TokenSequence<? extends TokenId> tokenSequence()
          Get token sequence of the top level language of the token hierarchy.

<T extends TokenId> TokenSequence<T> tokenSequence(Language<T> language)
          Get token sequence of the top level of the language hierarchy only if it's of the given language.

List<TokenSequence<? extends TokenId>> tokenSequenceList(LanguagePath languagePath, int startOffset, int endOffset)
          Get list of token sequences with the given language path from this hierarchy.

int tokenShiftEndOffset()
          Get end offset of the area where the tokens in the token hierarchy snapshot have explicitly shifted offsets.

int tokenShiftStartOffset()
          Get start offset of the area where the tokens in the token hierarchy snapshot have explicitly shifted offsets.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from class java.lang.Object
`clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

Method Detail

get

public static <D extends Document> TokenHierarchy<D> get(D doc)

Get or create mutable token hierarchy for the given swing document.
All the operations with the obtained token hierarchy must be done under document's read lock (or write lock).

Parameters:: doc - swing text document for which the token hiearchy should be obtained.
Returns:: token hierarchy or null in case the token hierarchy does not exist yet and the Language.class document property was not yet initialized with the valid language so the hierarchy cannot be created.

create

public static TokenHierarchy<Void> create(CharSequence inputText,
                                          Language<? extends TokenId> language)

Create token hierarchy for the given non-mutating input text (for example java.lang.String).

See Also:: create(CharSequence,boolean,Language,Set,InputAttributes)

create

public static <T extends TokenId> TokenHierarchy<Void> create(CharSequence inputText,
                                                              boolean copyInputText,
                                                              Language<T> language,
                                                              Set<T> skipTokenIds,
                                                              InputAttributes inputAttributes)

Create token hierarchy for the given input text.

Parameters:: inputText - input text containing the characters to tokenize.; copyInputText - true in case the content of the input will not be modified in the future so the created tokens can reference it.
false means that the text can change in the future and the tokens should not directly reference it. Instead copy of the necessary text from the input should be made and the original text should not be referenced.; language - language defining how the input will be tokenized.; skipTokenIds - set containing the token ids for which the tokens should not be created in the created token hierarchy.
null may be passed which means that no tokens will be skipped.
This applies to top level of the token hierarchy only (not to embedded tokens).
The provided set should be efficient enough - ideally created by e.g. Language.tokenCategoryMembers(String) or Language.merge(Collection,Collection).; inputAttributes - additional properties related to the input that may influence token creation or lexer operation for the particular language (such as version of the language to be used).
Returns:: non-null token hierarchy.

create

public static <T extends TokenId> TokenHierarchy<Void> create(Reader inputReader,
                                                              Language<T> language,
                                                              Set<T> skipTokenIds,
                                                              InputAttributes inputAttributes)

Create token hierarchy for the given reader.

Parameters:: inputReader - input reader containing the characters to tokenize.; language - language defining how the input will be tokenized.; skipTokenIds - set containing the token ids for which the tokens should not be created in the created token hierarchy.
null may be passed which means that no tokens will be skipped.
This applies to top level of the token hierarchy only (not to embedded tokens).
The provided set should be efficient enough - ideally created by e.g. Language.tokenCategoryMembers(String) or Language.merge(Collection,Collection).; inputAttributes - additional properties related to the input that may influence token creation or lexer operation for the particular language (such as version of the language to be used).
Returns:: non-null token hierarchy.

tokenSequence

public TokenSequence<? extends TokenId> tokenSequence()

Get token sequence of the top level language of the token hierarchy.
The token sequences for inner levels of the token hierarchy can be obtained by calling TokenSequence.embedded().

Returns:: non-null token sequence of the top level of the token hierarchy.

tokenSequence

public <T extends TokenId> TokenSequence<T> tokenSequence(Language<T> language)

Get token sequence of the top level of the language hierarchy only if it's of the given language.

Returns:: non-null token sequence or null if the top level token sequence satisfies the condition (tokenSequence().language() == language). Null is returned otherwise.

tokenSequenceList

public List<TokenSequence<? extends TokenId>> tokenSequenceList(LanguagePath languagePath,
                                                                int startOffset,
                                                                int endOffset)

Get list of token sequences with the given language path from this hierarchy.

Parameters:: languagePath - non-null language path that the obtained token sequences will all have.; startOffset - starting offset of the TSs to get. Use 0 for no limit. If the particular TS ends after this offset then it will be returned.; endOffset - ending offset of the TS to get. Use Integer.MAX_VALUE for no limit. If the particular TS starts before this offset then it will be returned.

languagePaths

public Set<LanguagePath> languagePaths()

Get a set of language paths used by this token hierarchy.
The set includes "static" paths that are those reachable by traversing token ids of the top language and searching for the default embeddings that could be created by LanguageHierarchy.embedding(Token,LanguagePath,InputAttributes).

isMutable

public boolean isMutable()

Whether input text of this token hierarchy is mutable or not.

Returns:: true if the input text is mutable or false otherwise.

mutableInputSource

public I mutableInputSource()

Get mutable input source providing text over which this token hierarchy was constructed.
For example it may be a swing text document instance Document in case the token hierarchy was constructed for its text.
Snapshot will return the same input source as the original mutable token hierarchy.

Returns:: mutable input source or null in case this token hierarchy was not created over mutable input source.

isActive

public boolean isActive()

Token hierarchy may be set inactive to release resources consumed by tokens.
Only token hierarchies over a mutable input can become inactive.

Returns:: true if valid tokens exist for this hierarchy or false if the token hierarchy is inactive and there are currently no active tokens to represent it.

addTokenHierarchyListener

public void addTokenHierarchyListener(TokenHierarchyListener listener)

Add listener for token changes inside this hierarchy.

Parameters:: listener - token change listener to be added.

removeTokenHierarchyListener

public void removeTokenHierarchyListener(TokenHierarchyListener listener)

Remove listener for token changes inside this hierarchy.

Parameters:: listener - token change listener to be removed.

createSnapshot

public TokenHierarchy<I> createSnapshot()

Create a snapshot of the present mutable token hierarchy.
Even with subsequent modifications to the "live" token hierarchy the tokens of the snapshot will retain the original ids, texts and offsets.
The snapshot retains the original token instances that were present in the token hierarchy at time of its creation.

The snapshot creation is cheap. With subsequent modifications of the mutable input source the snapshot maintenance brings an overhead. Therefore the snapshot should be released as soon as it's no longer needed. Ideally the releasing should be performed by using snapshotRelease(). Another way is to forget the reference to the snapshot token hierarchy but it depends on the garbage collector's releasing of the weak reference.
As the snapshot shares information with the live hierarchy its content must also be accessed under a read lock in the same way like the live hierarchy.
If a particular token in the snapshot is mutable then token.offset(snapshotHierarchy) will give the offset of the token in a snapshot while token.offset(null) will return the offset of the token in the live hierarchy.

The snapshot attempts to share tokens with the live token hierarchy.
Upon a first modification in the live token hierarchy (after the snapshot creation) an initial and ending areas of tokens shared between the snapshot and live hierarchy get created. The tokens that were in the live hierarchy prior to the modification (but which were removed from it because of the modification) are captured and used by the snapshot as the "middle" area. With subsequent modifications the initial and ending areas of shared tokens may be reduced (and the original tokens captured for the snapshot) if any of the tokens contained in them get modified.

The overhead of the subsequent token modifications for an existing snapshot in the present implementation are the following:

Removed token's text must be maintained which creates an overhead equal to the original token's text characters plus about 24 bytes.
This is a single-time overhead per each removed token referenced by at least one snapshot.
Token's original offset must be maintained. The overhead is about 32 bytes per token per snapshot.

Returns:: non-null new token hierarchy which is a snapshot of this token hierarchy. For non-mutable token hierarchies this method returns null (original token hierarchy may be used in the same way like the snapshot would be used).

isSnapshot

public boolean isSnapshot()

Check whether this token hierarchy is a snapshot.

Returns:: true if this is snapshot or false if not.

snapshotRelease

public void snapshotRelease()

Release snapshot - should only be called if this token hierarchy is a snapshot.

Throws:: IllegalStateException - if this token hierarchy was already released or it's not a snapshot.

isSnapshotReleased

public boolean isSnapshotReleased()

Check whether this snapshot is released.

Returns:: true if this snapshot is already released or false if not.
Throws:: IllegalStateException - if this token hierarchy is not a snapshot.

snapshotOf

public TokenHierarchy<I> snapshotOf()

If this token hierarchy is snapshot then return the token hierarchy for which this snapshot was constructed.

Returns:: live token hierarchy or null if this is not a snapshot.

tokenShiftStartOffset

public int tokenShiftStartOffset()

Get start offset of the area where the tokens in the token hierarchy snapshot have explicitly shifted offsets.
With subsequent modifications the area where the token offsets are shifted explicitly gets extended (modifications with lowest and highest offsets define the area boundaries).
Below this offset the snapshot uses all the tokens from the live token hierarchy directly.
Above this area (and below tokenShiftEndOffset() the tokens are either removed from the live token hierarchy or still present in it but all of them have explicitly corrected offsets.
The clients may get a token from the snapshot and check its offset to find out whether it's below token shift start offset.
If so then the token is present in the live token hierarchy as well and it has the same offset there like in the snapshot.

See Also:: tokenShiftEndOffset()

tokenShiftEndOffset

public int tokenShiftEndOffset()

Get end offset of the area where the tokens in the token hierarchy snapshot have explicitly shifted offsets.
The clients may get a token from the snapshot and check its offset to find out whether it's above token shift end offset.
If so then the token is present in the live token hierarchy as well and its offset there can be determined by using Token.offset(null).

See Also:: tokenShiftStartOffset()

Overview

Package

Class

Use

Tree

Deprecated

Index

Help

org.netbeans.modules.lexer/2 1.19.0 1

PREV CLASS NEXT CLASS

FRAMES NO FRAMES

SUMMARY: NESTED | FIELD | CONSTR | METHOD

DETAIL: FIELD | CONSTR | METHOD

TokenHierarchy (Lexer) - NetBeans API Javadoc (Current Development Version)

org.netbeans.api.lexer Class TokenHierarchy<I>

get

create

create

create

tokenSequence

tokenSequence

tokenSequenceList

languagePaths

isMutable

mutableInputSource

isActive

addTokenHierarchyListener

removeTokenHierarchyListener

createSnapshot

isSnapshot

snapshotRelease

isSnapshotReleased

snapshotOf

tokenShiftStartOffset

tokenShiftEndOffset

org.netbeans.api.lexer
Class TokenHierarchy<I>