LLMUnity.TokenSplitter Class Reference

Class implementing a token-based splitter. More...

Inheritance diagram for LLMUnity.TokenSplitter:

Public Member Functions
override async Task< List<(int, int)> >	Split (string input)
	Splits the provided phrase into chunks of a specific number of tokens (defined by the numTokens variable)
Public Member Functions inherited from LLMUnity.Chunking
void	ReturnChunks (bool returnChunks)
	Set to true to return chunks or the direct input with the Search function.
Task< List<(int, int)> >	Split (string input)
	Splits the provided phrase into chunks.
override string	Get (int key)
	Retrieves the phrase with the specific id.
override async Task< int >	Add (string inputString, string group="")
	Adds a phrase to the search after splitting it into chunks.
override void	Remove (int key)
	Removes a phrase and the phrase chunks from the search.
override int	Remove (string inputString, string group="")
	Removes a phrase and the phrase chunks from the search.
override int	Count ()
	Returns a count of the phrases.
override int	Count (string group)
	Returns a count of the phrases in a specific data group.
override async Task< int >	IncrementalSearch (string queryString, string group="")
	Allows to do search and retrieve results in batches (incremental search).
override ValueTuple< int[], float[], bool >	IncrementalFetchKeys (int fetchKey, int k)
	Retrieves the most similar search results in batches (incremental search). The phrase/chunk keys and distances are retrieved, as well as a parameter that dictates whether the search is exhausted. The returnChunks variable defines whether to return chunks or phrases.
override ValueTuple< string[], float[], bool >	IncrementalFetch (int fetchKey, int k)
	Retrieves the most similar search results in batches (incremental search). The phrases/chunks and their distances are retrieved, as well as a parameter that dictates whether the search is exhausted. The returnChunks variable defines whether to return chunks or phrases.
override void	IncrementalSearchComplete (int fetchKey)
	Completes the search and clears the cached results for an incremental search.
override void	Clear ()
	Clears the object and the associated search object.
Public Member Functions inherited from LLMUnity.SearchPlugin
void	SetSearch (SearchMethod search)
	Sets the search method of the plugin.
Public Member Functions inherited from LLMUnity.Searchable
string	Get (int key)
	Retrieves the phrase with the specific id.
Task< int >	Add (string inputString, string group="")
	Adds a phrase to the search.
int	Remove (string inputString, string group="")
	Removes a phrase from the search.
void	Remove (int key)
	Removes a phrase from the search.
int	Count ()
	Returns a count of the phrases.
int	Count (string group)
	Returns a count of the phrases in a specific data group.
void	Clear ()
	Clears the search object.
Task< int >	IncrementalSearch (string queryString, string group="")
	Allows to do search and retrieve results in batches (incremental search).
ValueTuple< int[], float[], bool >	IncrementalFetchKeys (int fetchKey, int k)
	Retrieves the most similar search results in batches (incremental search). The phrase keys and distances are retrieved, as well as a parameter that dictates whether the search is exhausted.
void	IncrementalSearchComplete (int fetchKey)
	Completes the search and clears the cached results for an incremental search.
async Task<(string[], float[])>	Search (string queryString, int k, string group="")
	Search for similar results to the provided query. The most similar results and their distances (dissimilarity) to the query are retrieved.
void	Save (string filePath)
	Saves the state of the search object.
async Task< bool >	Load (string filePath)
	Loads the state of the search object.

Public Attributes
int	numTokens = 10
	the number of tokens to split phrases into chunks

Detailed Description

Class implementing a token-based splitter.

Definition at line 14 of file TokenSplitter.cs.

Member Function Documentation

Split()

override async Task< List<(int, int)> > LLMUnity.TokenSplitter.Split ( string input )

inline

Splits the provided phrase into chunks of a specific number of tokens (defined by the numTokens variable)

Parameters

input

phrase

Returns

List of start/end indices of the split chunks

Definition at line 48 of file TokenSplitter.cs.

Member Data Documentation

numTokens

int LLMUnity.TokenSplitter.numTokens = 10

the number of tokens to split phrases into chunks

Definition at line 17 of file TokenSplitter.cs.

---

The documentation for this class was generated from the following file:

• Runtime/RAG/TokenSplitter.cs