[vc_row][vc_column][vc_headings linewidth=”0″ borderwidth=”1″ borderclr=”#000000″ title=”Audiocraft” google_fonts=”font_family:Comfortaa%3A300%2Cregular%2C700|font_style:700%20bold%20regular%3A700%3Anormal” titlesize=”60″ titleclr=”#000000″]Controllable Music Generation[/vc_headings][vc_single_image image=”3056″ alignment=”center”][vc_column_text]Audiocraft is a PyTorch library for audio generation research. It contains MusicGen, a controllable text-to-music model. MusicGen is a Transformer that generates 4 codebooks sampled at 50Hz. Unlike MusicLM, MusicGen generates all codebooks in 1 pass with a small delay, needing only 50 autoregressive steps/sec.[/vc_column_text][/vc_column][/vc_row][vc_row][vc_column][vc_separator style=”double” border_width=”5″][vc_headings style=”theme4″ borderclr=”#000000″ style2=”image” title=”What is MusicGen?” google_fonts=”font_family:Comfortaa%3A300%2Cregular%2C700|font_style:700%20bold%20regular%3A700%3Anormal” lineheight=”2″ titlesize=”40″ titleclr=”#000000″ image_id=”3479″][/vc_headings][vc_column_text css=”.vc_custom_1686498075285{margin-top: 30px !important;}”]MusicGen is a model for music generation that Audiocraft provides the code and models. It is a single-stage auto-regressive Transformer model that uses a 32kHz EnCodec tokenizer with 4 codebooks sampled at 50 Hz. MusicGen doesn’t need a self-supervised semantic representation like MusicLM, and it generates all 4 codebooks at once. By adding a small delay between the codebooks, MusicGen can parallelize their prediction, and reduce the auto-regressive steps to 50 per second of audio.[/vc_column_text][vc_separator][/vc_column][/vc_row][vc_row][vc_column][vc_headings style=”theme4″ borderclr=”#000000″ style2=”image” title=”Getting Started” google_fonts=”font_family:Comfortaa%3A300%2Cregular%2C700|font_style:700%20bold%20regular%3A700%3Anormal” lineheight=”3″ titlesize=”40″ titleclr=”#000000″ image_id=”2854″][/vc_headings][vc_column_text]
Installation
To use Audiocraft, you need Python 3.9, PyTorch 2.0.0, and a GPU with at least 16 GB of memory (for the medium-sized model). You can install Audiocraft by running this command:
# Best to make sure you have torch installed first, in particular before installing xformers.
# Don't run this if you already have PyTorch installed.
pip install 'torch>=2.0'
# Then proceed to one of the following
pip install -U audiocraft # stable release
pip install -U git+https://git@github.com/facebookresearch/audiocraft#egg=audiocraft # bleeding edge
pip install -e . # or if you cloned the repo locally
Usage
At Audiocraft, various modes of engagement with MusicGen are available:
- To try MusicGen, you have two options: run the jupyter notebook at
demo.ipynbon your own machine, or use the provided colab notebook. - To launch the gradio demo locally, run
python app.py. - A demo is also available on the
facebook/MusicGenHuggingFace Space . - Finally, apply a Gradio demo with a Colab GPU, follow the steps from @camenduru Colab.
API
Audiocraft offers a simple API and 4 ready-made models. The models are:
small: 300M model, text to music only – 🤗 Hubmedium: 1.5B model, text to music only – 🤗 Hubmelody: 1.5B model, text to music and text+melody to music – 🤗 Hublarge: 3.3B model, text to music only – 🤗 Hub
The medium or melody model gives the optimal balance of quality and computing. MusicGen requires a GPU to run locally. We suggest 16GB of memory, but you can generate short sequences or use the small model with less memory.
[/vc_column_text][vc_message]Newer versions of torchaudio require ffmpeg as a dependency. To install it, use the following command:
apt-get install ffmpeg
[/vc_message][vc_column_text]Below is a brief demonstration of how to use the API.
import torchaudio
from audiocraft.models import MusicGen
from audiocraft.data.audio import audio_write
model = MusicGen.get_pretrained('melody')
model.set_generation_params(duration=8) # generate 8 seconds.
wav = model.generate_unconditional(4) # generates 4 unconditional audio samples
descriptions = ['happy rock', 'energetic EDM', 'sad jazz']
wav = model.generate(descriptions) # generates 3 samples.
melody, sr = torchaudio.load('./assets/bach.mp3')
# generates using the melody from the given audio and the provided descriptions.
wav = model.generate_with_chroma(descriptions, melody[None].expand(3, -1, -1), sr)
for idx, one_wav in enumerate(wav):
# Will save under {idx}.wav, with loudness normalization at -14 db LUFS.
audio_write(f'{idx}', one_wav.cpu(), model.sample_rate, strategy="loudness", loudness_compressor=True)
[/vc_column_text][/vc_column][/vc_row][vc_row][vc_column width=”1/3″][vc_btn title=”Project Page” color=”inverse” align=”center” i_align=”right” i_icon_fontawesome=”fab fa-github” add_icon=”true” link=”url:https%3A%2F%2Fgithub.com%2Ffacebookresearch%2Faudiocraft|target:_blank”][/vc_column][vc_column width=”1/3″][vc_btn title=”Huggin Face Page” color=”warning” align=”center” i_align=”right” i_icon_fontawesome=”far fa-laugh-beam” add_icon=”true” link=”url:https%3A%2F%2Fhuggingface.co%2Fspaces%2Ffacebook%2FMusicGen|target:_blank”][/vc_column][vc_column width=”1/3″][vc_btn title=”Colab Page” color=”peacoc” align=”center” i_align=”right” i_icon_fontawesome=”fab fa-google” add_icon=”true” link=”url:https%3A%2F%2Fcolab.research.google.com%2Fdrive%2F1-Xe9NCdIs2sCUbiSmwHXozK6AAhMm7_i%3Fusp%3Dsharing|target:_blank”][/vc_column][/vc_row]
0 Comments