Falcon-40B: The Most Powerful Open-source Model

[vc_row][vc_column][vc_headings linewidth=”0″ borderwidth=”1″ borderclr=”#000000″ title=”Falcon-40B” google_fonts=”font_family:Comfortaa%3A300%2Cregular%2C700|font_style:700%20bold%20regular%3A700%3Anormal” titlesize=”60″ titleclr=”#000000″]The Most Powerful Open-source Model[/vc_headings][vc_single_image image=”4240″ img_size=”full” alignment=”center”][vc_column_text]Falcon-40B is a large language model (LLM) and one of Falcon LLM models with 40 billion parameters trained on 1,000B tokens of web data and curated corpora. It was developed by Technology Innovation Institute (TII) in Abu Dhabi and open-sourced under the Apache 2.0 license. Falcon-40B features an architecture optimized for inference, with FlashAttention and multiquery. It outperforms other open-source LLMs such as GPT-3, LLaMA, StableLM, RedPajama, and MPT.[/vc_column_text][vc_separator][/vc_column][/vc_row][vc_row][vc_column][vc_headings style=”theme4″ borderclr=”#000000″ style2=”image” title=”Falcon-40B Features” google_fonts=”font_family:Comfortaa%3A300%2Cregular%2C700|font_style:700%20bold%20regular%3A700%3Anormal” lineheight=”3″ titlesize=”40″ titleclr=”#000000″ image_id=”2871″][/vc_headings][evc_icon_with_text type=”icon-left” icon_library=”fontawesome” title_tag=”” icon_fontawesome=”fab fa-github” title=”Best open-source model” text=”Falcon-40B outperforms other open-source models such as LLaMA, StableLM, RedPajama, and MPT. It is one of the top ranked projects in Huggingface OpenLLM Leaderboard.” text_color=”#0a0000″][evc_icon_with_text type=”icon-left” icon_library=”fontawesome” title_tag=”” icon_fontawesome=”fas fa-tachometer-alt” title=”Optimized architecture” text=”Falcon-40B uses FlashAttention and multiquery techniques to improve inference speed and efficiency .” text_color=”#0a0000″][evc_icon_with_text type=”icon-left” icon_library=”fontawesome” title_tag=”” icon_fontawesome=”fas fa-user-shield” title=”Permissive license” text=”Falcon-40B is released under the Apache 2.0 license, which allows unrestricted commercial use without any royalties or restrictions.” text_color=”#0a0000″][evc_icon_with_text type=”icon-left” icon_library=”fontawesome” title_tag=”” icon_fontawesome=”fas fa-language” title=”Multilingual capabilities” text=”Falcon-40B supports English, German, Spanish, French, and has limited capabilities in Italian, Portuguese, Polish, Dutch, Romanian, Czech, Swedish.” text_color=”#0a0000″][evc_icon_with_text type=”icon-left” icon_library=”fontawesome” title_tag=”” icon_fontawesome=”fas fa-balance-scale-left” title=”Data quality at scale” text=”A data pipeline that extracts high-quality content from the web using extensive filtering and deduplication.” text_color=”#0a0000″][vc_separator color=”sandy_brown” border_width=”3″][/vc_column][/vc_row][vc_row][vc_column][vc_headings style=”theme4″ borderclr=”#000000″ style2=”image” title=”Get Started with Falcon-40B” google_fonts=”font_family:Comfortaa%3A300%2Cregular%2C700|font_style:700%20bold%20regular%3A700%3Anormal” lineheight=”3″ titlesize=”40″ titleclr=”#000000″ image_id=”2854″][/vc_headings][vc_column_text]

from transformers import AutoTokenizer, AutoModelForCausalLM
import transformers
import torch

model = "tiiuae/falcon-40b"

tokenizer = AutoTokenizer.from_pretrained(model)
pipeline = transformers.pipeline(
    "text-generation",
    model=model,
    tokenizer=tokenizer,
    torch_dtype=torch.bfloat16,
    trust_remote_code=True,
    device_map="auto",
)
sequences = pipeline(
   "Girafatron is obsessed with giraffes, the most glorious animal on the face of this Earth. Giraftron believes all other animals are irrelevant when compared to the glorious majesty of the giraffe.\nDaniel: Hello, Girafatron!\nGirafatron:",
    max_length=200,
    do_sample=True,
    top_k=10,
    num_return_sequences=1,
    eos_token_id=tokenizer.eos_token_id,
)
for seq in sequences:
    print(f"Result: {seq['generated_text']}")

[/vc_column_text][vc_separator][/vc_column][/vc_row][vc_row][vc_column][vc_headings style=”theme4″ borderclr=”#000000″ style2=”image” title=”Training Details” google_fonts=”font_family:Comfortaa%3A300%2Cregular%2C700|font_style:700%20bold%20regular%3A700%3Anormal” lineheight=”3″ titlesize=”40″ titleclr=”#000000″ image_id=”3479″][/vc_headings][vc_column_text]

Falcon-40B used RefinedWeb, a web dataset with high-quality filtering and deduplication, to train Falcon-40B on 1,000B tokens. they also added curated corpora to enhance the dataset, some of which were based on The Pile (Gao et al., 2020).

[/vc_column_text][vc_column_text css=”.vc_custom_1686380516525{border-top-width: 2px !important;border-right-width: 2px !important;border-bottom-width: 2px !important;border-left-width: 2px !important;background-color: #dbdbdb !important;border-left-color: #000000 !important;border-left-style: solid !important;border-right-color: #000000 !important;border-right-style: solid !important;border-top-color: #000000 !important;border-top-style: solid !important;border-bottom-color: #000000 !important;border-bottom-style: solid !important;border-radius: 1px !important;}”]

[/vc_column_text][vc_column_text]

RefinedWeb-Europe is made of the following languages:

[/vc_column_text][vc_column_text css=”.vc_custom_1686380543357{border-top-width: 2px !important;border-right-width: 2px !important;border-bottom-width: 2px !important;border-left-width: 2px !important;background-color: #dbdbdb !important;border-left-color: #000000 !important;border-left-style: solid !important;border-right-color: #000000 !important;border-right-style: solid !important;border-top-color: #000000 !important;border-top-style: solid !important;border-bottom-color: #000000 !important;border-bottom-style: solid !important;border-radius: 1px !important;}”]

[/vc_column_text][vc_column_text]

The data was tokenized with the Falcon-7B/40B tokenizer.

Training Procedure

Using 3D parallelism (TP=8, PP=4, DP=12) and ZeRO, Falcon-40B was  trained on 384 A100 40GB GPUs.

Training Hyperparameters

[/vc_column_text][vc_column_text css=”.vc_custom_1686380571919{border-top-width: 2px !important;border-right-width: 2px !important;border-bottom-width: 2px !important;border-left-width: 2px !important;background-color: #dbdbdb !important;border-left-color: #000000 !important;border-left-style: solid !important;border-right-color: #000000 !important;border-right-style: solid !important;border-top-color: #000000 !important;border-top-style: solid !important;border-bottom-color: #000000 !important;border-bottom-style: solid !important;border-radius: 1px !important;}”]

[/vc_column_text][vc_column_text]

How much it takes to train Falcon-40B?

Training started in December 2022 and took two months.

[/vc_column_text][vc_separator][/vc_column][/vc_row][vc_row][vc_column][vc_headings style=”theme4″ borderclr=”#000000″ style2=”image” title=”Technical Specifications” google_fonts=”font_family:Comfortaa%3A300%2Cregular%2C700|font_style:700%20bold%20regular%3A700%3Anormal” lineheight=”3″ titlesize=”40″ titleclr=”#000000″ image_id=”2854″][/vc_headings][vc_column_text]

Falcon-40B is a decoder-only model that learns to generate the next token in a sequence. It is based on the GPT-3 architecture (Brown et al., 2020), with some modifications:

• Rotary positional embeddings (Su et al., 2021) to encode the relative positions of tokens;
• Multiquery attention (Shazeer et al., 2019) and FlashAttention (Dao et al., 2022) to efficiently compute attention scores;
• Parallel attention/MLP decoder blocks with two layer normalization steps to stabilize the training.

Each tensor parallel degree has its own key and value in Falcon-40B’s multiquery, which uses a special version internally.[/vc_column_text][vc_column_text css=”.vc_custom_1686381161163{border-top-width: 2px !important;border-right-width: 2px !important;border-bottom-width: 2px !important;border-left-width: 2px !important;background-color: #dbdbdb !important;border-left-color: #000000 !important;border-left-style: solid !important;border-right-color: #000000 !important;border-right-style: solid !important;border-top-color: #000000 !important;border-top-style: solid !important;border-bottom-color: #000000 !important;border-bottom-style: solid !important;border-radius: 1px !important;}”]

[/vc_column_text][/vc_column][/vc_row][vc_row][vc_column][vc_separator color=”turquoise” border_width=”2″][vc_headings style=”theme4″ borderclr=”#000000″ style2=”image” title=”Lamitations” google_fonts=”font_family:Comfortaa%3A300%2Cregular%2C700|font_style:700%20bold%20regular%3A700%3Anormal” lineheight=”3″ titlesize=”40″ titleclr=”#000000″ image_id=”3031″][/vc_headings][vc_message]Falcon-40B is a multilingual model that can handle English, German, Spanish, French, and some other languages to a lesser extent. However, it is not suitable for languages outside its training data. Moreover, it may reflect the online prejudices and biases that are present in its large-scale web-based corpus.[/vc_message][/vc_column][/vc_row][vc_row][vc_column width=”1/3″][vc_btn title=”Visit Website” color=”warning” align=”center” i_align=”right” i_icon_fontawesome=”fas fa-external-link-alt” add_icon=”true” link=”url:https%3A%2F%2Ffalconllm.tii.ae%2F|target:_blank”][/vc_column][vc_column width=”1/3″][vc_btn title=”Project Page” color=”inverse” align=”center” i_align=”right” i_icon_fontawesome=”fab fa-github” add_icon=”true” link=”url:https%3A%2F%2Fgithub.com%2FDecentralised-AI%2Ffalcon-40b|target:_blank”][/vc_column][vc_column width=”1/3″][vc_btn title=”Huggin Face Page” color=”pink” align=”center” i_align=”right” i_icon_fontawesome=”far fa-grin-beam” add_icon=”true” link=”url:https%3A%2F%2Fhuggingface.co%2Ftiiuae%2Ffalcon-40b|target:_blank”][/vc_column][/vc_row][vc_row][vc_column][vc_separator][vc_headings style=”theme4″ borderclr=”#000000″ style2=”image” title=”FAQ” google_fonts=”font_family:Comfortaa%3A300%2Cregular%2C700|font_style:700%20bold%20regular%3A700%3Anormal” lineheight=”1.5″ titlesize=”40″ titleclr=”#000000″ image_id=”2875″][/vc_headings][vc_separator][vc_toggle title=”What is Falcon-40B and what can it do?” color=”juicy_pink”]Falcon-40B is a 40 billion parameters causal decoder-only model built by TII and trained on 1,000 billion tokens of RefinedWeb enhanced with curated corpora. It can generate text for various tasks such as summarization, text generation, chatbot, etc.[/vc_toggle][vc_separator color=”mulled_wine” border_width=”2″][vc_toggle title=”How can I use Falcon-40B?” color=”juicy_pink”]You can use Falcon-40B with the Hugging Face Transformers library. You need to install PyTorch 2.0 and import the AutoTokenizer and AutoModelForCausalLM classes from the transformers module. Then you can load the model and the tokenizer with the name “tiiuae/falcon-40b” and use the pipeline function to generate text with a given prompt.[/vc_toggle][vc_separator color=”mulled_wine” border_width=”2″][vc_toggle title=”What are the advantages of Falcon-40B?” color=”juicy_pink”]Falcon-40B is the best open-source model currently available. It outperforms other models such as LLaMA, StableLM, RedPajama, MPT, etc. on the OpenLLM Leaderboard. It also features an architecture optimized for inference, with FlashAttention and multiquery . It is made available under a permissive Apache 2.0 license allowing for commercial use, without any royalties or restrictions.[/vc_toggle][vc_separator color=”mulled_wine” border_width=”2″][vc_toggle title=”What is the difference between Falcon-40B and Falcon-40B-Instruct?” color=”juicy_pink”]Falcon-40B is a raw, pre-trained model that can be further finetuned for specific use cases. Falcon-40B-Instruct is a version of Falcon-40B that has been finetuned on a chat dataset and can take generic instructions in a chat format.[/vc_toggle][vc_separator color=”mulled_wine” border_width=”2″][/vc_column][/vc_row]

Comments

comments