[vc_row][vc_column][vc_headings linewidth=”0″ borderwidth=”1″ borderclr=”#000000″ title=”ChatGLM2-6B” google_fonts=”font_family:Comfortaa%3A300%2Cregular%2C700|font_style:700%20bold%20regular%3A700%3Anormal” titlesize=”60″ titleclr=”#000000″][/vc_headings][vc_single_image image=”3056″ alignment=”center”][vc_column_text]ChatGLM 2-6B is an open-source Chinese-English bilingual dialogue model that builds on the original ChatGLM-6B. It inherits the advantages of the first-generation model, such as fluent conversation and easy deployment.[/vc_column_text][vc_separator][/vc_column][/vc_row][vc_row][vc_column][vc_headings style=”theme4″ borderclr=”#000000″ style2=”image” title=”Features” google_fonts=”font_family:Comfortaa%3A300%2Cregular%2C700|font_style:700%20bold%20regular%3A700%3Anormal” lineheight=”3″ titlesize=”40″ titleclr=”#000000″ image_id=”2871″][/vc_headings][evc_icon_with_text type=”icon-left” icon_library=”fontawesome” title_tag=”” icon_fontawesome=”fas fa-database” title=”Improved base model” text=”GLM hybrid objective, 1.4T pre-training, human preference alignment, better MMLU, CEval, GSM8K, BBH scores” text_color=”#0a0000″][evc_icon_with_text type=”icon-left” icon_library=”fontawesome” title_tag=”” icon_fontawesome=”far fa-file-alt” title=”Longer context” text=”FlashAttention, 2K pedestal, 8K dialogue, limited single-round long document understanding” text_color=”#0a0000″][evc_icon_with_text type=”icon-left” icon_library=”fontawesome” title_tag=”” icon_fontawesome=”fas fa-microchip” title=”More efficient inference” text=”Multi-Query Attention, 42% faster inference, INT4 quantization, 6G memory supports 8K dialogue” text_color=”#0a0000″][evc_icon_with_text type=”icon-left” icon_library=”fontawesome” title_tag=”” icon_fontawesome=”fab fa-github-square” title=”More open protocol” text=”Academic and commercial use (with permission), donation welcome for ChatGLM3″ text_color=”#0a0000″][vc_separator color=”sandy_brown” border_width=”3″][vc_column_text]
valuation results
Using typical English and Chinese datasets, we evaluated ChatGLM2-6B on MMLU (English), C-Eval (Chinese), GSM8K (mathematics), and BBH (English). See evaluation for C-Eval scripts.
MMLU
[/vc_column_text][vc_raw_html css=”.vc_custom_1687875770809{border-top-width: 1px !important;border-right-width: 1px !important;border-bottom-width: 1px !important;border-left-width: 1px !important;background-color: #cecece !important;border-left-color: #000000 !important;border-left-style: solid !important;border-right-color: #000000 !important;border-right-style: solid !important;border-top-color: #000000 !important;border-top-style: solid !important;border-bottom-color: #000000 !important;border-bottom-style: solid !important;border-radius: 1px !important;}”]JTNDdGFibGUlM0UlMEElM0N0aGVhZCUzRSUwQSUzQ3RyJTNFJTBBJTNDdGglMjBzdHlsZSUzRCUyMmZvbnQtd2VpZ2h0JTNBYm9sZCUzQiUyMGNvbG9yJTNBYmxhY2slM0IlMjIlM0VNb2RlbCUzQyUyRnRoJTNFJTBBJTNDdGglMjBzdHlsZSUzRCUyMmZvbnQtd2VpZ2h0JTNBYm9sZCUzQiUyMGNvbG9yJTNBYmxhY2slM0IlMjIlM0VBdmVyYWdlJTNDJTJGdGglM0UlMEElM0N0aCUyMHN0eWxlJTNEJTIyZm9udC13ZWlnaHQlM0Fib2xkJTNCJTIwY29sb3IlM0FibGFjayUzQiUyMiUzRVNURU0lM0MlMkZ0aCUzRSUwQSUzQ3RoJTIwc3R5bGUlM0QlMjJmb250LXdlaWdodCUzQWJvbGQlM0IlMjBjb2xvciUzQWJsYWNrJTNCJTIyJTNFU29jaWFsJTIwU2NpZW5jZXMlM0MlMkZ0aCUzRSUwQSUzQ3RoJTIwc3R5bGUlM0QlMjJmb250LXdlaWdodCUzQWJvbGQlM0IlMjBjb2xvciUzQWJsYWNrJTNCJTIyJTNFSHVtYW5pdGllcyUzQyUyRnRoJTNFJTBBJTNDdGglMjBzdHlsZSUzRCUyMmZvbnQtd2VpZ2h0JTNBYm9sZCUzQiUyMGNvbG9yJTNBYmxhY2slM0IlMjIlM0VPdGhlcnMlM0MlMkZ0aCUzRSUwQSUzQyUyRnRyJTNFJTBBJTNDJTJGdGhlYWQlM0UlMEElM0N0Ym9keSUzRSUwQSUzQ3RyJTNFJTBBJTNDdGQlM0VDaGF0R0xNLTZCJTNDJTJGdGQlM0UlMEElM0N0ZCUzRTQwLjYzJTNDJTJGdGQlM0UlMEElM0N0ZCUzRTMzLjg5JTNDJTJGdGQlM0UlMEElM0N0ZCUzRTQ0Ljg0JTNDJTJGdGQlM0UlMEElM0N0ZCUzRTM5LjAyJTNDJTJGdGQlM0UlMEElM0N0ZCUzRTQ1LjcxJTNDJTJGdGQlM0UlMEElM0MlMkZ0ciUzRSUwQSUzQ3RyJTNFJTBBJTNDdGQlM0VDaGF0R0xNMi02QiUyMCUyOGJhc2UlMjklM0MlMkZ0ZCUzRSUwQSUzQ3RkJTNFNDcuODYlM0MlMkZ0ZCUzRSUwQSUzQ3RkJTNFNDEuMjAlM0MlMkZ0ZCUzRSUwQSUzQ3RkJTNFNTQuNDQlM0MlMkZ0ZCUzRSUwQSUzQ3RkJTNFNDMuNjYlM0MlMkZ0ZCUzRSUwQSUzQ3RkJTNFNTQuNDYlM0MlMkZ0ZCUzRSUwQSUzQyUyRnRyJTNFJTBBJTNDdHIlM0UlMEElM0N0ZCUzRUNoYXRHTE0yLTZCJTNDJTJGdGQlM0UlMEElM0N0ZCUzRTQ1LjQ2JTNDJTJGdGQlM0UlMEElM0N0ZCUzRTQwLjA2JTNDJTJGdGQlM0UlMEElM0N0ZCUzRTUxLjYxJTNDJTJGdGQlM0UlMEElM0N0ZCUzRTQxLjIzJTNDJTJGdGQlM0UlMEElM0N0ZCUzRTUxLjI0JTNDJTJGdGQlM0UlMEElM0MlMkZ0ciUzRSUwQSUzQyUyRnRib2R5JTNFJTBBJTNDJTJGdGFibGUlM0UlMEE=[/vc_raw_html][vc_message message_box_color=”black”]The Chat model was tested using the zero-shot CoT (Chain-of-Thought) method, and the Base model was tested using the few-shot answer-only method[/vc_message][vc_column_text]
C-Eval
[/vc_column_text][vc_raw_html css=”.vc_custom_1687875937819{border-top-width: 1px !important;border-right-width: 1px !important;border-bottom-width: 1px !important;border-left-width: 1px !important;background-color: #cecece !important;border-left-color: #000000 !important;border-left-style: solid !important;border-right-color: #000000 !important;border-right-style: solid !important;border-top-color: #000000 !important;border-top-style: solid !important;border-bottom-color: #000000 !important;border-bottom-style: solid !important;border-radius: 1px !important;}”]JTNDdGFibGUlM0UlMEElM0N0aGVhZCUzRSUwQSUzQ3RyJTNFJTBBJTNDdGglMjBzdHlsZSUzRCUyMmZvbnQtd2VpZ2h0JTNBYm9sZCUzQiUyMGNvbG9yJTNBYmxhY2slM0IlMjIlM0VNb2RlbCUzQyUyRnRoJTNFJTBBJTNDdGglMjBzdHlsZSUzRCUyMmZvbnQtd2VpZ2h0JTNBYm9sZCUzQiUyMGNvbG9yJTNBYmxhY2slM0IlMjIlM0VBdmVyYWdlJTNDJTJGdGglM0UlMEElM0N0aCUyMHN0eWxlJTNEJTIyZm9udC13ZWlnaHQlM0Fib2xkJTNCJTIwY29sb3IlM0FibGFjayUzQiUyMiUzRVNURU0lM0MlMkZ0aCUzRSUwQSUzQ3RoJTIwc3R5bGUlM0QlMjJmb250LXdlaWdodCUzQWJvbGQlM0IlMjBjb2xvciUzQWJsYWNrJTNCJTIyJTNFU29jaWFsJTIwU2NpZW5jZXMlM0MlMkZ0aCUzRSUwQSUzQ3RoJTIwc3R5bGUlM0QlMjJmb250LXdlaWdodCUzQWJvbGQlM0IlMjBjb2xvciUzQWJsYWNrJTNCJTIyJTNFSHVtYW5pdGllcyUzQyUyRnRoJTNFJTBBJTNDdGglMjBzdHlsZSUzRCUyMmZvbnQtd2VpZ2h0JTNBYm9sZCUzQiUyMGNvbG9yJTNBYmxhY2slM0IlMjIlM0VPdGhlcnMlM0MlMkZ0aCUzRSUwQSUzQyUyRnRyJTNFJTBBJTNDJTJGdGhlYWQlM0UlMEElM0N0Ym9keSUzRSUwQSUzQ3RyJTNFJTBBJTNDdGQlM0VDaGF0R0xNLTZCJTNDJTJGdGQlM0UlMEElM0N0ZCUzRTM4LjklM0MlMkZ0ZCUzRSUwQSUzQ3RkJTNFMzMuMyUzQyUyRnRkJTNFJTBBJTNDdGQlM0U0OC4zJTNDJTJGdGQlM0UlMEElM0N0ZCUzRTQxLjMlM0MlMkZ0ZCUzRSUwQSUzQ3RkJTNFMzguMCUzQyUyRnRkJTNFJTBBJTNDJTJGdHIlM0UlMEElM0N0ciUzRSUwQSUzQ3RkJTNFQ2hhdEdMTTItNkIlMjAlMjhiYXNlJTI5JTNDJTJGdGQlM0UlMEElM0N0ZCUzRTUxLjclM0MlMkZ0ZCUzRSUwQSUzQ3RkJTNFNDguNiUzQyUyRnRkJTNFJTBBJTNDdGQlM0U2MC41JTNDJTJGdGQlM0UlMEElM0N0ZCUzRTUxLjMlM0MlMkZ0ZCUzRSUwQSUzQ3RkJTNFNDkuOCUzQyUyRnRkJTNFJTBBJTNDJTJGdHIlM0UlMEElM0N0ciUzRSUwQSUzQ3RkJTNFQ2hhdEdMTTItNkIlM0MlMkZ0ZCUzRSUwQSUzQ3RkJTNFNTAuMSUzQyUyRnRkJTNFJTBBJTNDdGQlM0U0Ni40JTNDJTJGdGQlM0UlMEElM0N0ZCUzRTYwLjQlM0MlMkZ0ZCUzRSUwQSUzQ3RkJTNFNTAuNiUzQyUyRnRkJTNFJTBBJTNDdGQlM0U0Ni45JTNDJTJGdGQlM0UlMEElM0MlMkZ0ciUzRSUwQSUzQyUyRnRib2R5JTNFJTBBJTNDJTJGdGFibGUlM0UlMEE=[/vc_raw_html][vc_column_text]
GSM8K
[/vc_column_text][vc_raw_html css=”.vc_custom_1687876166505{border-top-width: 1px !important;border-right-width: 1px !important;border-bottom-width: 1px !important;border-left-width: 1px !important;background-color: #cecece !important;border-left-color: #000000 !important;border-left-style: solid !important;border-right-color: #000000 !important;border-right-style: solid !important;border-top-color: #000000 !important;border-top-style: solid !important;border-bottom-color: #000000 !important;border-bottom-style: solid !important;border-radius: 1px !important;}”]JTNDdGFibGUlM0UlMEElM0N0aGVhZCUzRSUwQSUzQ3RyJTNFJTBBJTNDdGglMjBzdHlsZSUzRCUyMmZvbnQtd2VpZ2h0JTNBYm9sZCUzQiUyMGNvbG9yJTNBYmxhY2slM0IlMjIlM0VNb2RlbCUzQyUyRnRoJTNFJTBBJTNDdGglMjBzdHlsZSUzRCUyMmZvbnQtd2VpZ2h0JTNBYm9sZCUzQiUyMGNvbG9yJTNBYmxhY2slM0IlMjIlM0VBY2N1cmFjeSUzQyUyRnRoJTNFJTBBJTNDdGglMjBzdHlsZSUzRCUyMmZvbnQtd2VpZ2h0JTNBYm9sZCUzQiUyMGNvbG9yJTNBYmxhY2slM0IlMjIlM0VBY2N1cmFjeSUyMCUyOENoaW5lc2UlMjklMkElM0MlMkZ0aCUzRSUwQSUzQyUyRnRyJTNFJTBBJTNDJTJGdGhlYWQlM0UlMEElM0N0Ym9keSUzRSUwQSUzQ3RyJTNFJTBBJTNDdGQlM0VDaGF0R0xNLTZCJTNDJTJGdGQlM0UlMEElM0N0ZCUzRTQuODIlM0MlMkZ0ZCUzRSUwQSUzQ3RkJTNFNS44NSUzQyUyRnRkJTNFJTBBJTNDJTJGdHIlM0UlMEElM0N0ciUzRSUwQSUzQ3RkJTNFQ2hhdEdMTTItNkIlMjAlMjhiYXNlJTI5JTNDJTJGdGQlM0UlMEElM0N0ZCUzRTMyLjM3JTNDJTJGdGQlM0UlMEElM0N0ZCUzRTI4Ljk1JTNDJTJGdGQlM0UlMEElM0MlMkZ0ciUzRSUwQSUzQ3RyJTNFJTBBJTNDdGQlM0VDaGF0R0xNMi02QiUzQyUyRnRkJTNFJTBBJTNDdGQlM0UyOC4wNSUzQyUyRnRkJTNFJTBBJTNDdGQlM0UyMC40NSUzQyUyRnRkJTNFJTBBJTNDJTJGdHIlM0UlMEElM0MlMkZ0Ym9keSUzRSUwQSUzQyUyRnRhYmxlJTNFJTBB[/vc_raw_html][vc_message message_box_color=”black”]
All models were tested using the few-shot CoT method, and the CoT prompt came from http://arxiv.org/abs/2201.11903
* We translated 8 questions and CoT prompt in GSM500K using the Translate API and performed human proofreading
[/vc_message][vc_column_text]
BBH
[/vc_column_text][vc_raw_html css=”.vc_custom_1687876372169{border-top-width: 1px !important;border-right-width: 1px !important;border-bottom-width: 1px !important;border-left-width: 1px !important;background-color: #cecece !important;border-left-color: #000000 !important;border-left-style: solid !important;border-right-color: #000000 !important;border-right-style: solid !important;border-top-color: #000000 !important;border-top-style: solid !important;border-bottom-color: #000000 !important;border-bottom-style: solid !important;border-radius: 1px !important;}”]JTNDdGFibGUlM0UlMEElM0N0aGVhZCUzRSUwQSUzQ3RyJTNFJTBBJTNDdGglMjBzdHlsZSUzRCUyMmZvbnQtd2VpZ2h0JTNBYm9sZCUzQiUyMGNvbG9yJTNBYmxhY2slM0IlMjIlM0VNb2RlbCUzQyUyRnRoJTNFJTBBJTNDdGglMjBzdHlsZSUzRCUyMmZvbnQtd2VpZ2h0JTNBYm9sZCUzQiUyMGNvbG9yJTNBYmxhY2slM0IlMjIlM0VBY2N1cmFjeSUzQyUyRnRoJTNFJTBBJTNDJTJGdHIlM0UlMEElM0MlMkZ0aGVhZCUzRSUwQSUzQ3Rib2R5JTNFJTBBJTNDdHIlM0UlMEElM0N0ZCUzRUNoYXRHTE0tNkIlM0MlMkZ0ZCUzRSUwQSUzQ3RkJTNFMTguNzMlM0MlMkZ0ZCUzRSUwQSUzQyUyRnRyJTNFJTBBJTNDdHIlM0UlMEElM0N0ZCUzRUNoYXRHTE0yLTZCJTIwJTI4YmFzZSUyOSUzQyUyRnRkJTNFJTBBJTNDdGQlM0UzMy42OCUzQyUyRnRkJTNFJTBBJTNDJTJGdHIlM0UlMEElM0N0ciUzRSUwQSUzQ3RkJTNFQ2hhdEdMTTItNkIlM0MlMkZ0ZCUzRSUwQSUzQ3RkJTNFMzAuMDAlM0MlMkZ0ZCUzRSUwQSUzQyUyRnRyJTNFJTBBJTNDJTJGdGJvZHklM0UlMEElM0MlMkZ0YWJsZSUzRSUwQQ==[/vc_raw_html][/vc_column][/vc_row][vc_row][vc_column][vc_headings style=”theme4″ borderclr=”#000000″ style2=”image” title=”How It Works” google_fonts=”font_family:Comfortaa%3A300%2Cregular%2C700|font_style:700%20bold%20regular%3A700%3Anormal” lineheight=”3″ titlesize=”40″ titleclr=”#000000″ image_id=”2854″][/vc_headings][vc_column_text]
Environment installation
To get started, clone this repository:
git clone https://github.com/THUDM/ChatGLM2-6B
cd ChatGLM2-6B
To install the required libraries, run pip with the following command, specifying the recommended version or higher for optimal inference speed: pip install -r requirements.txttransformers4.30.2torch
[/vc_column_text][vc_column_text]
Load the model locally
To run the code, you need to download the model from Hugging Face Hub. This is a full model that the code will use automatically. If you have a slow or unstable internet connection, you might face issues or delays in downloading the model. A possible solution is to download the model to your local machine and then load it from there.transformers
To get a model from Hugging Face Hub, you need to set up Git LFS and run it.
git clone https://huggingface.co/THUDM/chatglm2-6b
A faster way to get models from Hugging Face Hub is to download only the model code, without the checkpoints.
GIT_LFS_SKIP_SMUDGE=1 git clone https://huggingface.co/THUDM/chatglm2-6b
To update the model parameter file, follow this link and save the file to your local directory, overwriting the existing one.chatglm2-6b
o use the model from your local folder, you need to change the code above with the local folder path where you saved the model.THUDM/chatglm2-6bchatglm2-6b
The model implementation may change over time. To ensure compatibility with the model implementation invariant, you can increase the argument when you invoke. is the most recent version number, see Change Log for all versions.from_pretrainedrevision="v1.0"v1.0
[/vc_column_text][/vc_column][/vc_row][vc_row][vc_column][vc_column_text]
Web Demo
First, install Gradio: then run the web_demo.py in the repository:pip install gradio
python web_demo.py
The output address of the program, which operates a Web server, can be opened in a browser for functionality.
[/vc_column_text][vc_message message_box_color=”black”]The default mode is a startup, which does not create a public network link. To enable Internet access, change the setting from share=Falsetoshare=True[/vc_message][vc_column_text]
@AdamBear deserves our gratitude for creating the web demo with Streamlit. To run it, you have to install these extra dependencies first:web_demo2.py
pip install streamlit streamlit-chat
Use this command to launch it:
streamlit run web_demo2.py
The web demo works better with Streamlit when the input prompt is longer.
[/vc_column_text][/vc_column][/vc_row][vc_row][vc_column][vc_column_text]
Command line Demo
Run the cli_demo.py in the repository:
python cli_demo.py
The program will use the command line to communicate with the user. The user can type commands and press enter to get a response from the program. To reset the conversation history, the user can press enter without typing anything. To exit the program, the user can type terminate and press enter.clearstop
[/vc_column_text][/vc_column][/vc_row][vc_row][vc_column][vc_column_text]
API deployment
Before running the api.py file, make sure you have installed all the required packages from the repository:pip install fastapi uvicorn
python api.py
The POST method is used to call the deployment, which runs on port 8000 on-premises by default.
curl -X POST "http://127.0.0.1:8000" \
-H 'Content-Type: application/json' \
-d '{"prompt": "你好", "history": []}'
The resulting return value is
{
"response":"你好👋！我是人工智能助手 ChatGLM2-6B，很高兴见到你，欢迎问我任何问题。",
"history":[["你好","你好👋！我是人工智能助手 ChatGLM2-6B，很高兴见到你，欢迎问我任何问题。"]],
"status":200,
"time":"2023-03-23 21:38:40"
}
We appreciate @hiyouga’s contribution to the OpenAI-formatted streaming API deployment, which enables any ChatGPT-based application, such as ChatGPT-Next-Web. to use it as a backend. To deploy, simply run openai_api.py in your repository:
python openai_api.py
This is the sample code for making an API call:
import openai
if __name__ == "__main__":
openai.api_base = "http://localhost:8000/v1"
openai.api_key = "none"
for chunk in openai.ChatCompletion.create(
model="chatglm2-6b",
messages=[
{"role": "user", "content": "你好"}
],
stream=True
):
if hasattr(chunk.choices[0].delta, "content"):
print(chunk.choices[0].delta.content, end="", flush=True)
Low-cost deployment
Model quantization
To reduce the video memory usage, you can load the model with quantization instead of FP16 precision. The following code shows how to do this. The model needs about 13GB of video memory with FP16 precision.
# 按需修改，目前只支持 4/8 bit 量化
model = AutoModel.from_pretrained("THUDM/chatglm2-6b", trust_remote_code=True).quantize(8).cuda()
Reducing the model to 4 bits affects its performance, but ChatGLM2-6B can generate natural and smooth texts even with this quantization.
A quantized model can help you save memory. To use it, just load it directly:
model = AutoModel.from_pretrained("THUDM/chatglm2-6b-int4",trust_remote_code=True).cuda()
CPU deployment
You can perform inference on the CPU if you lack GPU hardware, but it will take longer. Follow these steps to use it (you need about 32GB RAM).
model = AutoModel.from_pretrained("THUDM/chatglm2-6b", trust_remote_code=True).float()
A quantized model can reduce memory consumption when the available memory is insufficient.
model = AutoModel.from_pretrained("THUDM/chatglm2-6b-int4",trust_remote_code=True).float()
To use the quantized model on the CPU, you need to install it. These are usually pre-installed on Linux systems. For Windows, make sure to select when you install TDM-GCC. Refer to Q1 on MacOS.gccopenmpopenmpgccTDM-GCC 10.3.0gcc 11.3.0
[/vc_column_text][/vc_column][/vc_row][vc_row][vc_column width=”1/4″][mvc_advanced_button align=”center” btn_text=”Github” icon_size=”25″ use_theme_fonts=”yes” btn_icon=”fab fa-github” btn_url=”url:https%3A%2F%2Fgithub.com%2FTHUDM%2FChatGLM2-6B|target:_blank” btn_clr=”#ffffff” btn_bg=”#0a0a0a” btn_radius=”50″][/vc_column][vc_column width=”1/4″][mvc_advanced_button align=”center” btn_text=”HF” icon_size=”25″ use_theme_fonts=”yes” btn_url=”url:https%3A%2F%2Fhuggingface.co%2FTHUDM%2Fchatglm2-6b|target:_blank” btn_clr=”#ffffff” btn_bg=”#ff4300″ btn_radius=”50″ btn_icon=”far fa-grin-beam”][/vc_column][vc_column width=”1/4″][mvc_advanced_button align=”center” btn_text=”Twitter” icon_size=”25″ use_theme_fonts=”yes” btn_icon=”fab fa-twitter” btn_url=”url:https%3A%2F%2Ftwitter.com%2Fthukeg|target:_blank” btn_clr=”#ffffff” btn_bg=”#00acee” btn_radius=”50″][/vc_column][vc_column width=”1/4″][mvc_advanced_button align=”center” btn_text=”Paper” icon_size=”25″ use_theme_fonts=”yes” btn_icon=”fas fa-copy” btn_url=”url:https%3A%2F%2Farxiv.org%2Fabs%2F2210.02414|target:_blank” btn_clr=”#ffffff” btn_bg=”#1e73be” btn_radius=”50″][/vc_column][/vc_row]
0 Comments