Text generation web UI

One-click-installer

The script uses Miniconda to set up a Conda environment in the installer_files folder.

If you ever need to install something manually in the installer_files environment, you can launch an interactive shell using the cmd script: cmd_linux.sh, cmd_windows.bat, cmd_macos.sh, or cmd_wsl.bat.

• There is no need to run any of those scripts (start_, update_wizard_, or cmd_) as admin/root.
• To install the requirements for extensions, you can use the extensions_reqs script for your OS. At the end, this script will install the main requirements for the project to make sure that they take precedence in case of version conflicts.
• For additional instructions about AMD and WSL setup, consult the documentation.
• For automated installation, you can use the GPU_CHOICE, USE_CUDA118, LAUNCH_AFTER_INSTALL, and INSTALL_EXTENSIONS environment variables. For instance: GPU_CHOICE=A USE_CUDA118=FALSE LAUNCH_AFTER_INSTALL=FALSE INSTALL_EXTENSIONS=TRUE ./start_linux.sh.

Manual installation using Conda

Recommended if you have some experience with the command-line.

0. Install Conda

https://docs.conda.io/en/latest/miniconda.html

On Linux or WSL, it can be automatically installed with these two commands (source):

curl -sL "https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh" > "Miniconda3.sh"
bash Miniconda3.sh

1. Create a new conda environment

conda create -n textgen python=3.11
conda activate textgen

2. Install Pytorch

The up-to-date commands can be found here: https://pytorch.org/get-started/locally/.

For NVIDIA, you also need to install the CUDA runtime libraries:

conda install -y -c "nvidia/label/cuda-12.1.1" cuda-runtime

If you need nvcc to compile some library manually, replace the command above with

conda install -y -c "nvidia/label/cuda-12.1.1" cuda

3. Install the web UI

git clone https://github.com/oobabooga/text-generation-webui
cd text-generation-webui
pip install -r <requirements file according to table below>

Requirements file to use:

Start the web UI

conda activate textgen
cd text-generation-webui
python server.py

Then browse to

http://localhost:7860/?__theme=dark

AMD GPU on Windows

1. Use requirements_cpu_only.txt or requirements_cpu_only_noavx2.txt in the command above.

2. Manually install llama-cpp-python using the appropriate command for your hardware: Installation from PyPI.

   • Use the LLAMA_HIPBLAS=on toggle.
   • Note the Windows remarks.

3. Manually install AutoGPTQ: Installation.

   • Perform the from-source installation - there are no prebuilt ROCm packages for Windows.

Older NVIDIA GPUs

1. For Kepler GPUs and older, you will need to install CUDA 11.8 instead of 12:

pip3 install torch==2.2.1 torchvision==0.17.1 torchaudio==2.2.1 --index-url https://download.pytorch.org/whl/cu118
conda install -y -c "nvidia/label/cuda-11.8.0" cuda-runtime

2. bitsandbytes >= 0.39 may not work. In that case, to use --load-in-8bit, you may have to downgrade like this:
   • Linux: pip install bitsandbytes==0.38.1
   • Windows: pip install https://github.com/jllllll/bitsandbytes-windows-webui/raw/main/bitsandbytes-0.38.1-py3-none-any.whl

Manual install

The requirements*.txt above contain various wheels precompiled through GitHub Actions. If you wish to compile things manually, or if you need to because no suitable wheels are available for your hardware, you can use requirements_nowheels.txt and then install your desired loaders manually.

Alternative: Docker

For NVIDIA GPU:
ln -s docker/{nvidia/Dockerfile,nvidia/docker-compose.yml,.dockerignore} .
For AMD GPU: 
ln -s docker/{amd/Dockerfile,intel/docker-compose.yml,.dockerignore} .
For Intel GPU:
ln -s docker/{intel/Dockerfile,amd/docker-compose.yml,.dockerignore} .
For CPU only
ln -s docker/{cpu/Dockerfile,cpu/docker-compose.yml,.dockerignore} .
cp docker/.env.example .env
#Create logs/cache dir : 
mkdir -p logs cache
# Edit .env and set: 
#   TORCH_CUDA_ARCH_LIST based on your GPU model
#   APP_RUNTIME_GID      your host user's group id (run `id -g` in a terminal)
#   BUILD_EXTENIONS      optionally add comma separated list of extensions to build
# Edit CMD_FLAGS.txt and add in it the options you want to execute (like --listen --cpu)
# 
docker compose up --build

• You need to have Docker Compose v2.17 or higher installed. See this guide for instructions.
• For additional docker files, check out this repository.

Updating the requirements

From time to time, the requirements*.txt change. To update, use these commands:

conda activate textgen
cd text-generation-webui
pip install -r <requirements file that you have used> --upgrade

Basic settings

Model loader

Accelerate/transformers

bitsandbytes 4-bit

⚠️ Requires minimum compute of 7.0 on Windows at the moment.

llama.cpp

ExLlamav2

AutoGPTQ

GPTQ-for-LLaMa

HQQ

DeepSpeed

RoPE (for llama.cpp, ExLlamaV2, and transformers)

Gradio

API

Multimodal