LLM多模态•audiocraft•av(interfacing FFmpeg API)•Audio/Video/Bitstream•pytorch•sklearn•numpy•pandas•spacy•librosa•opencv
无论是ChatGPT、 LLM大语言模型、还是Meta公司的AI生成音乐🎶，
都需要对 Audio、Video、Bitstream 进行处理。
Text的算法库 SpaCy, numpy, pytorch/Tensorflow/Transformers,…
Audio的算法库 librosa, numpy, pytorch/Tensorflow/Transformers, …；
Image/Video的算法库 Pillow, OpenCV, numpy, pytorch/Tensorflow/Transformers, …;
以Meta(Facebook已改名为Meta)开源的 audiocraft 为例：
- ASR(Audio转文本, 人机语音交互与识别)、
- TTS(文本合成语音)、
- NLP(自然语言处理)、
- NLG(自然语言生成)、
- Content Generation(智能生成 Text/Image/Audio/Video/…)
audiocraft 的:
- NLP 部分用的是Python库SpaCy;
- audio/video 部分用的是Python库 av(用 Cython 封装好FFmpeg C/C++ API)，极大的方便 Audio/Video/Bitstream 的上层应用例如 AI/MachinLearning调用.
- 当然还可以参考Python的 OpenCV / av 库封装其它的多模态内容接口; 实现全媒体覆盖(Article/Text/Image/Audio/Video/…)
SpaCy: Industrial-Strength Natural Language Processing
https://spacy.io/
av 这个库(https://pypi.org/project/av/#description)
FFmpeg: https://ffmpeg.org/documentation.html
PyAV is a Pythonic binding for the [FFmpeg][ffmpeg] libraries.
We aim to provide all of the power and control of the underlying library, but manage the gritty details as much as possible.
PyAV is for direct and precise access to your media via containers, streams, packets, codecs, and frames.
It exposes a few transformations of that data, and helps you get your data to/from other packages (e.g. Numpy and Pillow).
This power does come with some responsibility as working with media is horrendously complicated and PyAV can't abstract it away or make all the best decisions for you.
If the `ffmpeg` command does the job without you bending over backwards, PyAV is likely going to be more of a hindrance than a help.
But where you can't work without it, PyAV is a critical tool.
Installation
------------
Due to the complexity of the dependencies, PyAV is not always the easiest Python package to install from source.
Since release 8.0.0 binary wheels are provided on [PyPI][pypi] for Linux, Mac and Windows linked against a modern FFmpeg.
You can install these wheels by running:
```bash
pip install av
```
If you want to use your existing FFmpeg, the source version of PyAV is on [PyPI][pypi] too:
```bash
pip install av --no-binary av
```
Alternative installation methods
--------------------------------
Another way of installing PyAV is via [conda-forge][conda-forge]:
```bash
conda install av -c conda-forge
```
https://github.com/abaelhe/audiocraft
Audiocraft is a PyTorch library for deep learning research on audio generation. At the moment, it contains the code for MusicGen, a state-of-the-art controllable text-to-music model.
MusicGen
Audiocraft provides the code and models for MusicGen, [a simple and controllable model for music generation][arxiv]. MusicGen is a single stage auto-regressive
Transformer model trained over a 32kHz EnCodec tokenizer with 4 codebooks sampled at 50 Hz. Unlike existing methods like MusicLM, MusicGen doesn't require a self-supervised semantic representation, and it generates
all 4 codebooks in one pass. By introducing a small delay between the codebooks, we show we can predict
them in parallel, thus having only 50 auto-regressive steps per second of audio.
Check out our [sample page][musicgen_samples] or test the available demo!
We use 20K hours of licensed music to train MusicGen. Specifically, we rely on an internal dataset of 10K high-quality music tracks, and on the ShutterStock and Pond5 music data.
Installation
Audiocraft requires Python 3.9, PyTorch 2.0.0, and a GPU with at least 16 GB of memory (for the medium-sized model). To install Audiocraft, you can run the following:
# Best to make sure you have torch installed first, in particular before installing xformers.
# Don't run this if you already have PyTorch installed.
pip install 'torch>=2.0'
# Then proceed to one of the following
pip install -U audiocraft # stable release
pip install -U git+https://git@github.com/facebookresearch/audiocraft#egg=audiocraft # bleeding edge
pip install -e . # or if you cloned the repo locally
</code>