Qwen2-Audio 4-bit

We adopt a 4-bit quantization strategy implemented through the bitsandbytes library. The model weights are loadedin 4-bit precision (load in 4bit=True). We use the NormalFloat4 (NF4) quantization format (bnb 4bit quant type="nf4"), as NF4 has been shown to provide better reconstruction fidelity than standard uniform INT4 quantization for large language models. To further decrease storage, we enable double quantization (bnb 4bit use double quant=True), where the quantization constants themselves are quantized. Computations during inference and adaptation are performed in half precision (bnb 4bit compute dtype=float16). Model size is 5.9GB.