Microsoft Unveils Phi-3.5: A Family of Efficient and Powerful Small Language Models

Microsoft's latest generation of Small Language Models (SLMs), the Phi-3.5 family, boasts superior performance across diverse benchmarks encompassing language, reasoning, coding, and mathematics. Designed for both power and efficiency, these models expand Azure's offerings, providing developers with enhanced tools for generative AI applications. Building on user feedback since the April 2024 Phi-3 launch, Phi-3.5 introduces three key models: Phi-3.5-mini, Phi-3.5-vision, and Phi-3.5-MoE (a Mixture-of-Experts model).

Key Model Features:

• Phi-3.5-mini: Features an extended 128K context length and improved multilingual capabilities.
• Phi-3.5-vision: Boasts enhanced multi-frame image comprehension and reasoning, leading to improved single-image benchmark results.
• Phi-3.5-MoE: A Mixture-of-Experts model leveraging 16 experts and 6.6B active parameters, outperforming larger models while maintaining efficiency, multilingual support, and robust safety features. It also supports a 128K context length.

Phi-3.5-MoE: A Deep Dive

The flagship Phi-3.5-MoE model comprises 16 experts, each with 3.8B parameters, totaling 42B parameters. However, only 6.6B parameters are active at any given time. This architecture surpasses comparable-sized dense models in performance and quality, supporting over 20 languages. Rigorous safety training, incorporating both proprietary and open-source data, employs Direct Preference Optimization (DPO) and Supervised Fine-Tuning (SFT) to ensure harmlessness and helpfulness.

Phi-3.5-MoE Training Data:

The model's training utilized 4.9 trillion tokens (10% multilingual) from diverse sources:

• High-quality, rigorously filtered public documents and educational data.
• Synthetic "textbook-like" data for math, coding, and reasoning skills.
• High-quality chat data reflecting human preferences for instruction following, truthfulness, and helpfulness.

The table above highlights Phi-3.5-MoE's superior performance compared to larger models across various benchmarks.

This table demonstrates Phi-3.5-MoE's strong multilingual capabilities, outperforming larger models on multilingual tasks.

Phi-3.5-mini: Small Size, Big Impact

Phi-3.5-mini benefits from additional pre-training and post-training (DPO, PPO, SFT) using multilingual and high-quality data.

Phi-3.5-mini Training Data:

Similar to Phi-3.5-MoE, Phi-3.5-mini's training data (3.4 trillion tokens) includes filtered public documents, synthetic data, and high-quality chat data.

This table illustrates Phi-3.5-mini's competitive performance against larger models.

This table showcases Phi-3.5-mini's improved multilingual performance, particularly in languages like Arabic, Dutch, and Finnish.

The 128K context length of Phi-3.5-mini makes it suitable for long-document processing tasks.

Phi-3.5-vision: Image Understanding Redefined

Phi-3.5-vision leverages a diverse training dataset, including filtered public documents, image-text data, synthetic data, and high-quality chat data. It excels in multi-frame image understanding, enabling tasks like image comparison and multi-image summarization. It also shows improved performance on single-image benchmarks.

These tables illustrate Phi-3.5-vision's performance improvements on multi-image benchmarks.

Trying Out the Models:

Instructions and examples are provided for using Phi-3.5-mini and Phi-3.5-vision via Hugging Face and Azure AI Studio. Note that Hugging Face Spaces was used for Phi-3.5-vision due to its GPU requirements.

Conclusion:

The Phi-3.5 family offers a compelling range of cost-effective, high-performance SLMs for both open-source developers and Azure users. Each model caters to specific needs, from the compact and multilingual Phi-3.5-mini to the powerful and versatile Phi-3.5-MoE and the image-focused Phi-3.5-vision.

Frequently Asked Questions: (Included in original text)

The above is the detailed content of What Makes Phi 3.5 SLMs a Game-Changer for Generative AI?. For more information, please follow other related articles on the PHP Chinese website!