LLMUnity.Chunking Class Reference

Class implementing the chunking functionality. More...

Inheritance diagram for LLMUnity.Chunking:

Public Member Functions
void	ReturnChunks (bool returnChunks)
	Set to true to return chunks or the direct input with the Search function.
Task< List<(int, int)> >	Split (string input)
	Splits the provided phrase into chunks.
override string	Get (int key)
	Retrieves the phrase with the specific id.
override async Task< int >	Add (string inputString, string group="")
	Adds a phrase to the search after splitting it into chunks.
override void	Remove (int key)
	Removes a phrase and the phrase chunks from the search.
override int	Remove (string inputString, string group="")
	Removes a phrase and the phrase chunks from the search.
override int	Count ()
	Returns a count of the phrases.
override int	Count (string group)
	Returns a count of the phrases in a specific data group.
override async Task< int >	IncrementalSearch (string queryString, string group="")
	Allows to do search and retrieve results in batches (incremental search).
override ValueTuple< int[], float[], bool >	IncrementalFetchKeys (int fetchKey, int k)
	Retrieves the most similar search results in batches (incremental search). The phrase/chunk keys and distances are retrieved, as well as a parameter that dictates whether the search is exhausted. The returnChunks variable defines whether to return chunks or phrases.
override ValueTuple< string[], float[], bool >	IncrementalFetch (int fetchKey, int k)
	Retrieves the most similar search results in batches (incremental search). The phrases/chunks and their distances are retrieved, as well as a parameter that dictates whether the search is exhausted. The returnChunks variable defines whether to return chunks or phrases.
override void	IncrementalSearchComplete (int fetchKey)
	Completes the search and clears the cached results for an incremental search.
override void	Clear ()
	Clears the object and the associated search object.
Public Member Functions inherited from LLMUnity.SearchPlugin
void	SetSearch (SearchMethod search)
	Sets the search method of the plugin.
Public Member Functions inherited from LLMUnity.Searchable
string	Get (int key)
	Retrieves the phrase with the specific id.
Task< int >	Add (string inputString, string group="")
	Adds a phrase to the search.
int	Remove (string inputString, string group="")
	Removes a phrase from the search.
void	Remove (int key)
	Removes a phrase from the search.
int	Count ()
	Returns a count of the phrases.
int	Count (string group)
	Returns a count of the phrases in a specific data group.
void	Clear ()
	Clears the search object.
Task< int >	IncrementalSearch (string queryString, string group="")
	Allows to do search and retrieve results in batches (incremental search).
ValueTuple< int[], float[], bool >	IncrementalFetchKeys (int fetchKey, int k)
	Retrieves the most similar search results in batches (incremental search). The phrase keys and distances are retrieved, as well as a parameter that dictates whether the search is exhausted.
void	IncrementalSearchComplete (int fetchKey)
	Completes the search and clears the cached results for an incremental search.
async Task<(string[], float[])>	Search (string queryString, int k, string group="")
	Search for similar results to the provided query. The most similar results and their distances (dissimilarity) to the query are retrieved.
void	Save (string filePath)
	Saves the state of the search object.
async Task< bool >	Load (string filePath)
	Loads the state of the search object.

Detailed Description

Class implementing the chunking functionality.

Definition at line 17 of file Chunking.cs.

Member Function Documentation

Add()

override async Task< int > LLMUnity.Chunking.Add ( string inputString, string group = "" )

inline

Adds a phrase to the search after splitting it into chunks.

Parameters

inputString	input phrase
group	data group to add it to

Returns

phrase id

Definition at line 63 of file Chunking.cs.

Clear()

override void LLMUnity.Chunking.Clear ( )

inline

Clears the object and the associated search object.

Definition at line 265 of file Chunking.cs.

Count() [1/2]

override int LLMUnity.Chunking.Count ( )

inline

Returns a count of the phrases.

Returns

phrase count

Definition at line 147 of file Chunking.cs.

Count() [2/2]

override int LLMUnity.Chunking.Count ( string group )

inline

Returns a count of the phrases in a specific data group.

Parameters

group

data group

Returns

phrase count

Definition at line 157 of file Chunking.cs.

Get()

override string LLMUnity.Chunking.Get ( int key )

inline

Retrieves the phrase with the specific id.

Parameters

key

phrase id

Returns

phrase

Definition at line 47 of file Chunking.cs.

Here is the caller graph for this function:

IncrementalFetch()

override ValueTuple< string[], float[], bool > LLMUnity.Chunking.IncrementalFetch ( int fetchKey, int k )

inlinevirtual

Retrieves the most similar search results in batches (incremental search). The phrases/chunks and their distances are retrieved, as well as a parameter that dictates whether the search is exhausted. The returnChunks variable defines whether to return chunks or phrases.

Parameters

fetchKey	incremental search key
k	number of results to retrieve

Returns

A tuple containing:

• Array of retrieved phrases/chunks (string[]).
• Array of distances for each result (float[]).
• bool indicating if the search is exhausted.

Reimplemented from LLMUnity.Searchable.

Definition at line 241 of file Chunking.cs.

IncrementalFetchKeys()

override ValueTuple< int[], float[], bool > LLMUnity.Chunking.IncrementalFetchKeys ( int fetchKey, int k )

inline

Retrieves the most similar search results in batches (incremental search). The phrase/chunk keys and distances are retrieved, as well as a parameter that dictates whether the search is exhausted. The returnChunks variable defines whether to return chunks or phrases.

Parameters

fetchKey	incremental search key
k	number of results to retrieve

Returns

A tuple containing:

• Array of retrieved keys (int[]).
• Array of distances for each result (float[]).
• bool indicating if the search is exhausted.

Definition at line 189 of file Chunking.cs.

Here is the caller graph for this function:

IncrementalSearch()

override async Task< int > LLMUnity.Chunking.IncrementalSearch ( string queryString, string group = "" )

inline

Allows to do search and retrieve results in batches (incremental search).

Parameters

queryString	search query
group	data group to search in

Returns

incremental search key

Definition at line 169 of file Chunking.cs.

IncrementalSearchComplete()

override void LLMUnity.Chunking.IncrementalSearchComplete ( int fetchKey )

inline

Completes the search and clears the cached results for an incremental search.

Parameters

fetchKey

incremental search key

Definition at line 257 of file Chunking.cs.

Here is the caller graph for this function:

Remove() [1/2]

override void LLMUnity.Chunking.Remove ( int key )

inline

Removes a phrase and the phrase chunks from the search.

Parameters

key

phrase id

Definition at line 97 of file Chunking.cs.

Here is the caller graph for this function:

Remove() [2/2]

override int LLMUnity.Chunking.Remove ( string inputString, string group = "" )

inline

Removes a phrase and the phrase chunks from the search.

Parameters

inputString	input phrase
group	data group to remove it from

Returns

number of removed phrases

Definition at line 130 of file Chunking.cs.

ReturnChunks()

void LLMUnity.Chunking.ReturnChunks ( bool returnChunks )

inline

Set to true to return chunks or the direct input with the Search function.

Parameters

returnChunks

whether to return chunks

Definition at line 30 of file Chunking.cs.

Here is the caller graph for this function:

Split()

Task< List<(int, int)> > LLMUnity.Chunking.Split ( string input )

abstract

Splits the provided phrase into chunks.

Parameters

input

phrase

Returns

List of start/end indices of the split chunks

Here is the caller graph for this function:

---

The documentation for this class was generated from the following file:

• Runtime/RAG/Chunking.cs