LLM_runtime.h File Reference

Runtime loading and management of LLM libraries. More...

#include <fstream>
#include <sstream>
#include <vector>
#include <iostream>
#include <setjmp.h>
#include <type_traits>
#include <algorithm>
#include <cstdlib>
#include "defs.h"
#include "error_handling.h"
#include "LLM.h"
#include <dlfcn.h>
#include <unistd.h>
#include <limits.h>

Include dependency graph for LLM_runtime.h:

This graph shows which files directly or indirectly include this file:

Go to the source code of this file.

Classes
class	LLMService
	Runtime loader for LLM libraries. More...

Macros
#define	LOAD_LIB(path)   dlopen(path, RTLD_LAZY)
	Load library macro for Unix.
#define	GET_SYM(handle, name)   dlsym(handle, name)
	Get symbol macro for Unix.
#define	CLOSE_LIB(handle)   dlclose(handle)
	Close library macro for Unix.
#define	LLM_FUNCTIONS_LIST(M)
	Macro defining the list of dynamically loaded LLM functions.
#define	DECLARE_FN(name, ret, ...)    ret (*name)(__VA_ARGS__) = nullptr;
	Declare function pointers for dynamically loaded functions.

Typedefs
using	LibHandle = void *
	Unix library handle type.

Functions
const std::string	os_library_dir ()
	Get OS-specific library directory.
const std::vector< std::string >	available_architectures (bool gpu)
	Get available architectures for the platform.
std::vector< std::string >	get_default_library_env_vars ()
	Get default environment variables for library paths.
const char *	Available_Architectures (bool gpu)
	Get available architectures (C API)

Detailed Description

Runtime loading and management of LLM libraries.

Provides dynamic library loading capabilities for LLM backends, architecture detection, and cross-platform library management

Definition in file LLM_runtime.h.

Macro Definition Documentation

CLOSE_LIB

#define CLOSE_LIB

(

handle

)

dlclose(handle)

Close library macro for Unix.

Definition at line 45 of file LLM_runtime.h.

DECLARE_FN

#define DECLARE_FN	(		name,
			ret,
			... )    ret (*name)(__VA_ARGS__) = nullptr;

Declare function pointers for dynamically loaded functions.

Uses the LLM_FUNCTIONS_LIST macro to declare all required function pointers

Definition at line 213 of file LLM_runtime.h.

GET_SYM

#define GET_SYM	(		handle,
			name )   dlsym(handle, name)

Get symbol macro for Unix.

Definition at line 44 of file LLM_runtime.h.

LLM_FUNCTIONS_LIST

#define LLM_FUNCTIONS_LIST

(

M

)

Value:

    M(LLMService_Registry, void, LLMProviderRegistry *)                                                           \
    M(LLMService_InjectErrorState, void, ErrorState *)                                                            \
    M(LLMService_Construct, LLMProvider *, const char *, int, int, int, bool, int, int, bool, int, const char **) \
    M(LLMService_From_Command, LLMProvider *, const char *)

Macro defining the list of dynamically loaded LLM functions.

Parameters

M	Macro to apply to each function signature

This macro is used to generate function pointer declarations and loading code

Definition at line 53 of file LLM_runtime.h.

LOAD_LIB

#define LOAD_LIB

(

path

)

dlopen(path, RTLD_LAZY)

Load library macro for Unix.

Definition at line 43 of file LLM_runtime.h.

Typedef Documentation

LibHandle

using LibHandle = void *

Unix library handle type.

Definition at line 42 of file LLM_runtime.h.

Function Documentation

available_architectures()

const std::vector< std::string > available_architectures

(

bool

gpu

)

Get available architectures for the platform.

Parameters

gpu	Whether to include GPU-enabled architectures

Returns

Vector of available architecture strings

Detects available CPU/GPU architectures for library selection

Definition at line 23 of file LLM_runtime.cpp.

Here is the caller graph for this function:

get_default_library_env_vars()

std::vector< std::string > get_default_library_env_vars

(

)

Get default environment variables for library paths.

Returns

Vector of environment variable names to check for library paths

Returns platform-specific environment variables used for library loading

Definition at line 108 of file LLM_runtime.cpp.

Here is the caller graph for this function:

os_library_dir()

const std::string os_library_dir

(

)

Get OS-specific library directory.

Returns

Path to platform-specific library directory

Returns the appropriate library directory for the current operating system