Python API?

[54601]

Sorry, I'm new to this....

Thank to the one-click script, I have successfully installed rkllm, and can run 'rkllm xxx.rkllm 4096 4096' with decent response.

However, I don't know if there's any python api, so I can build my own python script with it?

[Disorn1998]

Hi
Yes, there is a Python API that you can use to build your own scripts. Typically, when you install rkllm using a one-click script, the necessary Python library components are installed along with it.

You can try a simple script like the one below to get started

import rkllm

#Load your specific rkllm model file
model = rkllm.load_model('xxx.rkllm')

#Define your desired model parameters
#You can adjust these values as needed
context_length = 4096
generate_length = 4096

#Generate a response from the model
response = model.generate(
prompt='What is the capital of France?',
max_context_length=context_length,
max_generate_length=generate_length
)

#Print the generated output
print(response)

rkllm.load_model('xxx.rkllm'): This is used to load your model file (in this case, xxx.rkllm).

model.generate(...): This is the main function for running the model. You can provide a prompt and specify parameters like max_context_length and max_generate_length just like you would on the command line.

You try frist!

[54601]

Hi! Thank you for reply!

My problem is, although I did use the one-click script to install rkllm (curl https://raw.githubusercontent.com/Pelochus/ezrknpu/main/install.sh | sudo bash), I don't have the module rkllm in python.

I run the script again and it completed without any complain, but I still can't find a module named rkllm in python.

[54601]

This is the output of the one-click script:

% Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  2922  100  2922    0     0      7      0  0:06:57  0:06:35  0:00:22   401

#########################################
Checking script arguments...
#########################################


#########################################
Checking root permission...
#########################################


#########################################
Installing apt dependencies...
#########################################

Get:1 https://packages.microsoft.com/repos/code stable InRelease [3590 B]
Hit:2 http://ppa.launchpad.net/jjriek/rockchip/ubuntu noble InRelease
Get:3 https://packages.microsoft.com/repos/code stable/main armhf Packages [20.1 kB]
Get:4 https://packages.microsoft.com/repos/code stable/main arm64 Packages [20.0 kB]
Hit:5 http://ports.ubuntu.com noble InRelease
Get:6 https://packages.microsoft.com/repos/code stable/main amd64 Packages [19.9 kB]
Get:7 http://ppa.launchpad.net/jjriek/rockchip-multimedia/ubuntu noble InRelease [17.8 kB]
Get:8 http://ports.ubuntu.com noble-updates InRelease [126 kB]
Get:9 https://esm.ubuntu.com/apps/ubuntu noble-apps-security InRelease [7613 B]
Hit:10 https://apt.v2raya.org v2raya InRelease
Get:11 https://esm.ubuntu.com/apps/ubuntu noble-apps-updates InRelease [7480 B]
Hit:12 https://ppa.launchpadcontent.net/ansible/ansible/ubuntu noble InRelease
Get:13 https://esm.ubuntu.com/infra/ubuntu noble-infra-security InRelease [7474 B]
Hit:15 https://ppa.launchpadcontent.net/jjriek/panfork-mesa/ubuntu noble InRelease
Get:16 https://ryanfortner.github.io/box64-debs/debian ./ InRelease [2115 B]
Get:17 https://esm.ubuntu.com/infra/ubuntu noble-infra-updates InRelease [7473 B]
Hit:18 https://ppa.launchpadcontent.net/qbittorrent-team/qbittorrent-stable/ubuntu noble InRelease
Get:19 https://repo.steampowered.com/steam stable InRelease [2858 B]
Get:20 https://repo.jellyfin.org/ubuntu noble InRelease [10.6 kB]
Get:21 http://ports.ubuntu.com noble-backports InRelease [126 kB]
Get:22 https://deb.sagernet.org * InRelease
Hit:23 https://ryanfortner.github.io/box86-debs/debian ./ InRelease
Get:24 https://ryanfortner.github.io/box64-debs/debian ./ Packages [3131 B]
Get:25 http://ports.ubuntu.com noble-security InRelease [126 kB]
Get:26 https://dl.cloudsmith.io/public/moonlight-game-streaming/moonlight-qt/deb/ubuntu noble InRelease [5110 B]
Get:27 http://ports.ubuntu.com noble-updates/main arm64 Components [161 kB]
Get:28 http://ports.ubuntu.com noble-updates/universe arm64 Components [375 kB]
Get:29 http://ports.ubuntu.com noble-updates/restricted arm64 Components [212 B]
Get:30 http://ports.ubuntu.com noble-updates/multiverse arm64 Components [212 B]
Get:31 http://ports.ubuntu.com noble-backports/main arm64 Components [3564 B]
Get:32 http://ports.ubuntu.com noble-backports/universe arm64 Components [30.7 kB]
Get:33 http://ports.ubuntu.com noble-backports/restricted arm64 Components [216 B]
Get:34 http://ports.ubuntu.com noble-backports/multiverse arm64 Components [212 B]
Get:35 http://ports.ubuntu.com noble-security/main arm64 Components [18.4 kB]
Get:36 http://ports.ubuntu.com noble-security/universe arm64 Components [52.2 kB]
Get:37 http://ports.ubuntu.com noble-security/restricted arm64 Components [212 B]
Get:38 http://ports.ubuntu.com noble-security/multiverse arm64 Components [212 B]
Hit:39 https://download.docker.com/linux/ubuntu noble InRelease
Hit:14 https://packagecloud.io/github/git-lfs/ubuntu noble InRelease
Fetched 1170 kB in 13s (92.3 kB/s)
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
55 packages can be upgraded. Run 'apt list --upgradable' to see them.
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
git is already the newest version (1:2.43.0-1ubuntu7.3).
git-lfs is already the newest version (3.7.0).
python-is-python3 is already the newest version (3.11.4-1).
python3-pip is already the newest version (24.0+dfsg-1ubuntu1.2).
python3-dev is already the newest version (3.12.3-0ubuntu2).
build-essential is already the newest version (12.10ubuntu1).
cmake is already the newest version (3.28.3-1build7).
libxslt1-dev is already the newest version (1.1.39-0exp1ubuntu0.24.04.2).
zlib1g-dev is already the newest version (1:1.3.dfsg-3.1ubuntu2.1).
libglib2.0-dev is already the newest version (2.80.0-6ubuntu3.4).
libsm6 is already the newest version (2:1.2.3-1build3).
libprotobuf-dev is already the newest version (3.21.12-8.2ubuntu0.2).
libhdf5-dev is already the newest version (1.10.10+repack-3.1ubuntu4).
libgl1 is already the newest version (1.7.0-1build1).
libglx-mesa0 is already the newest version (1:23.0.5-0ubuntu1~panfork~git221210.120202c6757~noble6).
The following packages were automatically installed and are no longer required:
  libgit2-1.7 libhttp-parser2.9 libllvm17t64 libqrtr1 libssh2-1t64 libstd-rust-1.75 libstd-rust-dev protection-domain-mapper qrtr-tools
Use 'sudo apt autoremove' to remove them.
0 upgraded, 0 newly installed, 0 to remove and 55 not upgraded.

#########################################
Cloning main repo with submodules...
#########################################

Cloning into 'ezrknpu'...
remote: Enumerating objects: 140, done.
remote: Counting objects: 100% (47/47), done.
remote: Compressing objects: 100% (34/34), done.
remote: Total 140 (delta 23), reused 29 (delta 13), pack-reused 93 (from 1)
Receiving objects: 100% (140/140), 55.56 KiB | 235.00 KiB/s, done.
Resolving deltas: 100% (69/69), done.
Submodule 'ezrknn-llm' (https://github.com/Pelochus/ezrknn-llm) registered for path 'ezrknn-llm'
Submodule 'ezrknn-toolkit2' (https://github.com/Pelochus/ezrknn-toolkit2) registered for path 'ezrknn-toolkit2'
Cloning into '/home/user/ezrknpu/ezrknn-llm'...
remote: Enumerating objects: 1733, done.
remote: Counting objects: 100% (241/241), done.
remote: Compressing objects: 100% (89/89), done.
remote: Total 1733 (delta 187), reused 152 (delta 152), pack-reused 1492 (from 2)
Receiving objects: 100% (1733/1733), 251.99 MiB | 3.16 MiB/s, done.
Resolving deltas: 100% (615/615), done.
Cloning into '/home/user/ezrknpu/ezrknn-toolkit2'...
remote: Enumerating objects: 2872, done.
remote: Counting objects: 100% (110/110), done.
remote: Compressing objects: 100% (44/44), done.
remote: Total 2872 (delta 78), reused 66 (delta 66), pack-reused 2762 (from 3)
Receiving objects: 100% (2872/2872), 3.72 GiB | 1.10 MiB/s, done.
Resolving deltas: 100% (1030/1030), done.
Submodule path 'ezrknn-llm': checked out '3b1bb27d8aa8326bdc5409cae51cb86492263fa4'
Submodule path 'ezrknn-toolkit2': checked out '3780dd7e3f1b96f9f76533ac0bbcde1dd268c5ad'

#########################################
Installing RKNN LLM with install.sh script...
#########################################


#########################################
Checking root permission...
#########################################


#########################################
Installing RKNN LLM libraries...
#########################################


#########################################
Compiling LLM runtime for Linux...
#########################################

-- The C compiler identification is GNU 13.3.0
-- The CXX compiler identification is GNU 13.3.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/gcc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/g++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Configuring done (0.9s)
-- Generating done (0.0s)
-- Build files have been written to: /home/user/ezrknpu/ezrknn-llm/examples/DeepSeek-R1-Distill-Qwen-1.5B_Demo/deploy/build/build_linux_aarch64_Release
[ 50%] Building CXX object CMakeFiles/llm_demo.dir/src/llm_demo.cpp.o
[100%] Linking CXX executable llm_demo
[100%] Built target llm_demo

#########################################
Moving rkllm to /usr/bin...
#########################################


#########################################
Increasing file limit for all users (needed for LLMs to run)...
#########################################


#########################################
Done installing ezrknn-llm!
#########################################


#########################################
Installing RKNN Toolkit 2 with install.sh script...
#########################################


#########################################
Checking root permission...
#########################################


#########################################
Installing pip dependencies for ARM64...
#########################################

Processing ./rknn-toolkit-lite2/packages/rknn_toolkit_lite2-2.3.2-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Requirement already satisfied: numpy in /usr/local/lib/python3.12/dist-packages (from rknn-toolkit-lite2==2.3.2) (1.26.4)
Requirement already satisfied: psutil in /usr/local/lib/python3.12/dist-packages (from rknn-toolkit-lite2==2.3.2) (7.0.0)
Requirement already satisfied: ruamel.yaml in /usr/local/lib/python3.12/dist-packages (from rknn-toolkit-lite2==2.3.2) (0.18.14)
Requirement already satisfied: ruamel.yaml.clib>=0.2.7 in /usr/local/lib/python3.12/dist-packages (from ruamel.yaml->rknn-toolkit-lite2==2.3.2) (0.2.12)
rknn-toolkit-lite2 is already installed with the same version as the provided wheel. Use --force-reinstall to force an installation of the wheel.
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
Requirement already satisfied: protobuf<=4.25.4,>=4.21.0 in /usr/local/lib/python3.12/dist-packages (from -r ./rknn-toolkit2/packages/arm64/arm64_requirements_cp312.txt (line 4)) (4.25.4)
Requirement already satisfied: psutil>=5.9.0 in /usr/local/lib/python3.12/dist-packages (from -r ./rknn-toolkit2/packages/arm64/arm64_requirements_cp312.txt (line 7)) (7.0.0)
Requirement already satisfied: ruamel.yaml>=0.17.21 in /usr/local/lib/python3.12/dist-packages (from -r ./rknn-toolkit2/packages/arm64/arm64_requirements_cp312.txt (line 8)) (0.18.14)
Requirement already satisfied: scipy>=1.9.3 in /usr/local/lib/python3.12/dist-packages (from -r ./rknn-toolkit2/packages/arm64/arm64_requirements_cp312.txt (line 9)) (1.16.1)
Requirement already satisfied: tqdm>=4.64.1 in /usr/local/lib/python3.12/dist-packages (from -r ./rknn-toolkit2/packages/arm64/arm64_requirements_cp312.txt (line 10)) (4.67.1)
Requirement already satisfied: opencv-python>=4.5.5.64 in /usr/local/lib/python3.12/dist-packages (from -r ./rknn-toolkit2/packages/arm64/arm64_requirements_cp312.txt (line 11)) (4.11.0.86)
Requirement already satisfied: fast-histogram>=0.11 in /usr/local/lib/python3.12/dist-packages (from -r ./rknn-toolkit2/packages/arm64/arm64_requirements_cp312.txt (line 12)) (0.14)
Requirement already satisfied: Pillow>=10.0.1 in /usr/lib/python3/dist-packages (from -r ./rknn-toolkit2/packages/arm64/arm64_requirements_cp312.txt (line 13)) (10.2.0)
Requirement already satisfied: numpy<=1.26.4 in /usr/local/lib/python3.12/dist-packages (from -r ./rknn-toolkit2/packages/arm64/arm64_requirements_cp312.txt (line 14)) (1.26.4)
Requirement already satisfied: onnx==1.16.1 in /usr/local/lib/python3.12/dist-packages (from -r ./rknn-toolkit2/packages/arm64/arm64_requirements_cp312.txt (line 17)) (1.16.1)
Requirement already satisfied: onnxruntime>=1.17.0 in /usr/local/lib/python3.12/dist-packages (from -r ./rknn-toolkit2/packages/arm64/arm64_requirements_cp312.txt (line 18)) (1.22.1)
Requirement already satisfied: torch<=2.2.0,>=1.13.1 in /usr/local/lib/python3.12/dist-packages (from -r ./rknn-toolkit2/packages/arm64/arm64_requirements_cp312.txt (line 19)) (2.2.0)
Requirement already satisfied: ruamel.yaml.clib>=0.2.7 in /usr/local/lib/python3.12/dist-packages (from ruamel.yaml>=0.17.21->-r ./rknn-toolkit2/packages/arm64/arm64_requirements_cp312.txt (line 8)) (0.2.12)
Requirement already satisfied: coloredlogs in /usr/local/lib/python3.12/dist-packages (from onnxruntime>=1.17.0->-r ./rknn-toolkit2/packages/arm64/arm64_requirements_cp312.txt (line 18)) (15.0.1)
Requirement already satisfied: flatbuffers in /usr/local/lib/python3.12/dist-packages (from onnxruntime>=1.17.0->-r ./rknn-toolkit2/packages/arm64/arm64_requirements_cp312.txt (line 18)) (25.2.10)
Requirement already satisfied: packaging in /usr/lib/python3/dist-packages (from onnxruntime>=1.17.0->-r ./rknn-toolkit2/packages/arm64/arm64_requirements_cp312.txt (line 18)) (24.0)
Requirement already satisfied: sympy in /usr/local/lib/python3.12/dist-packages (from onnxruntime>=1.17.0->-r ./rknn-toolkit2/packages/arm64/arm64_requirements_cp312.txt (line 18)) (1.14.0)
Requirement already satisfied: filelock in /usr/local/lib/python3.12/dist-packages (from torch<=2.2.0,>=1.13.1->-r ./rknn-toolkit2/packages/arm64/arm64_requirements_cp312.txt (line 19)) (3.18.0)
Requirement already satisfied: typing-extensions>=4.8.0 in /usr/lib/python3/dist-packages (from torch<=2.2.0,>=1.13.1->-r ./rknn-toolkit2/packages/arm64/arm64_requirements_cp312.txt (line 19)) (4.10.0)
Requirement already satisfied: networkx in /usr/local/lib/python3.12/dist-packages (from torch<=2.2.0,>=1.13.1->-r ./rknn-toolkit2/packages/arm64/arm64_requirements_cp312.txt (line 19)) (3.5)
Requirement already satisfied: jinja2 in /usr/lib/python3/dist-packages (from torch<=2.2.0,>=1.13.1->-r ./rknn-toolkit2/packages/arm64/arm64_requirements_cp312.txt (line 19)) (3.1.2)
Requirement already satisfied: fsspec in /usr/local/lib/python3.12/dist-packages (from torch<=2.2.0,>=1.13.1->-r ./rknn-toolkit2/packages/arm64/arm64_requirements_cp312.txt (line 19)) (2025.7.0)
Requirement already satisfied: humanfriendly>=9.1 in /usr/local/lib/python3.12/dist-packages (from coloredlogs->onnxruntime>=1.17.0->-r ./rknn-toolkit2/packages/arm64/arm64_requirements_cp312.txt (line 18)) (10.0)
Requirement already satisfied: mpmath<1.4,>=1.1.0 in /usr/local/lib/python3.12/dist-packages (from sympy->onnxruntime>=1.17.0->-r ./rknn-toolkit2/packages/arm64/arm64_requirements_cp312.txt (line 18)) (1.3.0)
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv

#########################################
Installing RKNN NPU API...
#########################################


#########################################
Compiling RKNN Benchmark for RK3588...
#########################################

build-linux.sh -t rk3588 -a aarch64 -b Release
Using gcc and g++ by default...
===================================
TARGET_SOC=RK3588
TARGET_ARCH=aarch64
BUILD_TYPE=Release
BUILD_DIR=/home/user/ezrknpu/ezrknn-toolkit2/rknpu2/examples/rknn_benchmark/build/build_RK3588_linux_aarch64_Release
CC=/usr/bin/gcc
CXX=/usr/bin/g++
===================================
-- The C compiler identification is GNU 13.3.0
-- The CXX compiler identification is GNU 13.3.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/gcc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/g++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Configuring done (0.7s)
-- Generating done (0.0s)
-- Build files have been written to: /home/user/ezrknpu/ezrknn-toolkit2/rknpu2/examples/rknn_benchmark/build/build_RK3588_linux_aarch64_Release
[ 66%] Building CXX object CMakeFiles/rknn_benchmark.dir/src/cnpy/cnpy.cpp.o
[ 66%] Building CXX object CMakeFiles/rknn_benchmark.dir/src/rknn_benchmark.cpp.o
[100%] Linking CXX executable rknn_benchmark
[100%] Built target rknn_benchmark
[100%] Built target rknn_benchmark
Install the project...
-- Install configuration: "Release"
-- Installing: /home/user/ezrknpu/ezrknn-toolkit2/rknpu2/examples/rknn_benchmark/install/rknn_benchmark_Linux/./rknn_benchmark
-- Set non-toolchain portion of runtime path of "/home/user/ezrknpu/ezrknn-toolkit2/rknpu2/examples/rknn_benchmark/install/rknn_benchmark_Linux/./rknn_benchmark" to "lib"
-- Installing: /home/user/ezrknpu/ezrknn-toolkit2/rknpu2/examples/rknn_benchmark/install/rknn_benchmark_Linux/lib/librknnrt.so

#########################################
Done installing ezrknn-toolkit2!
#########################################


#########################################
Everything done!
#########################################