Supported Models

Currently, most models in Huggingface transformers are supported. All layers in the models listed below can be parallelized. They include vision models like ViT, CLIP and speech models like Wav2Vec2 as well as language models.

Fully Supported Models

  • ALBERT

  • BART

  • BARThez (=BERT)

  • BERT

  • BERTweet (=BERT)

  • BertJapanese (=BERT)

  • BertGeneration

  • Blenderbot

  • Blenderbot Samll

  • BORT (=BERT)

  • CamemBERT (=RoBERTa)

  • CLIP

  • CPM

  • CTRL

  • DeBERTa

  • DeBERTa-v2

  • DeiT

  • DETR

  • DialoGPT (=GPT2)

  • DistilBERT

  • DPR (=BERT)

  • ELECTRA

  • FlauBERT (=XLM)

  • FSMT

  • Funnel Transformer

  • herBERT (=RoBERTa)

  • I-BERT

  • LayoutLM

  • LED

  • Longformer

  • LUKE

  • LXMERT

  • MarianMT

  • M2M100

  • MBart

  • Mobile BERT

  • MPNet

  • MT5 (=T5)

  • Megatron BERT (=BERT)

  • Megatron GPT2 (=GPT2)

  • OpenAI GPT

  • OpenAI GPT2

  • GPTNeo

  • Hubert

  • Pegasus

  • PhoBERT (=RoBERTa)

  • Reformer

  • RetriBERT

  • RoBERTa

  • RoFormer

  • Speech2Text

  • T5

  • ByT5 (=T5)

  • TAPAS

  • TransformerXL

  • ViT

  • VisualBERT

  • Wav2Vec2

  • XLM

  • XLM-RoBERTa (=RoBERTa)

  • XLNet

  • XLSR-Wave2Vec2

Partly Supported or Unsupported Models

At present the following models are partly supported or not supported.

Partly Supported Models

  • BigBird

  • BigBirdPegasus

  • ConvBERT

  • ProphetNet

  • XLM-ProphetNet

Unsupported Models

  • SqueezeBERT

  • RAG