Multimodal input

Multimodal models process text, image, audio, and video simultaneously for cross-modal understanding and fusion analysis.

1. Overview

Multimodal models are large language models capable of processing multiple types of information (text, images, audio, video) simultaneously. SiliconFlow provides several powerful models supporting different modality combinations that can:

Visual Understanding: Understand image content, OCR, image description
Video Analysis: Extract video frames, understand video content, action recognition
Audio Processing: Speech recognition, audio content analysis
Multimodal Fusion: Comprehensive analysis of multiple media types simultaneously

2. Supported Models Overview

Model Series	Vision	Audio	Video	Key Features
Qwen3-Omni Series	✅	✅	✅	Full multimodal support, simultaneous audio/video processing
Qwen3-VL Series	✅	❌	✅	Visual+video understanding, no audio support
GLM Series	✅	❌	❌	Vision understanding only
Qwen2-VL Series	✅	❌	❌	Vision understanding only
DeepseekVL2 Series	✅	❌	❌	Vision understanding only
Step3	✅	❌	❌	Vision understanding only

View the current list of supported multimodal models in the Model Hub. Supported models may be adjusted, please refer to the actual platform display.

3. Usage Methods

All multimodal models are called through the /chat/completions interface using a standardized messages format, where content can contain different types of content parts.

3.1 Basic Message Format

{
    "role": "user",
    "content": [
        {
            "type": "text" | "image_url" | "audio_url" | "video_url",
            "[type]_url": {
                // Configuration for corresponding type
            }
        }
    ]
}

3.2 Common Parameter Description

Image Parameters (`image_url`)

url: Image URL or base64 encoded data
detail: Detail level (auto, low, high)

Video Parameters (`video_url`)

url: Video URL or base64 encoded data
detail: Detail level (auto, low, high)
max_frames: Maximum number of frames to extract
fps: Frames per second, final frame count is min(fps × T, max_frames)

Audio Parameters (`audio_url`)

url: Audio URL or base64 encoded data

4. Usage Examples

4.1 Visual Understanding

Image Analysis

{
    "role": "user",
    "content": [
        {
            "type": "image_url",
            "image_url": {
                "url": "https://example.com/image.jpg",
                "detail": "high"
            }
        },
        {
            "type": "text",
            "text": "Describe the content of this image"
        }
    ]
}

Multi-image Comparison

{
    "role": "user",
    "content": [
        {
            "type": "image_url",
            "image_url": {"url": "https://example.com/image1.jpg"}
        },
        {
            "type": "image_url",
            "image_url": {"url": "https://example.com/image2.jpg"}
        },
        {
            "type": "text",
            "text": "Compare the similarities and differences between these two images"
        }
    ]
}

4.2 Video Understanding

Basic Video Analysis

{
    "role": "user",
    "content": [
        {
            "type": "video_url",
            "video_url": {
                "url": "https://example.com/video.mp4",
                "detail": "high",
                "max_frames": 16,
                "fps": 1
            }
        },
        {
            "type": "text",
            "text": "Summarize the main content of this video"
        }
    ]
}

Multimodal Analysis (Video + Image)

{
    "role": "user",
    "content": [
        {
            "type": "video_url",
            "video_url": {
                "url": "https://example.com/video.mp4",
                "detail": "high",
                "max_frames": 16,
                "fps": 1
            }
        },
        {
            "type": "image_url",
            "image_url": {"url": "https://example.com/thumbnail.jpg"}
        },
        {
            "type": "text",
            "text": "Based on the video and thumbnail, analyze the theme and target audience of this video"
        }
    ]
}

4.3 Audio Understanding

Audio Content Analysis

{
    "role": "user",
    "content": [
        {
            "type": "audio_url",
            "audio_url": {
                "url": "https://example.com/audio.mp3"
            }
        },
        {
            "type": "text",
            "text": "Transcribe the content of this audio"
        }
    ]
}

4.4 Full Modality Analysis

Audio-Video Comprehensive Analysis

{
    "role": "user",
    "content": [
        {
            "type": "video_url",
            "video_url": {
                "url": "https://example.com/video.mp4",
                "detail": "high",
                "max_frames": 16,
                "fps": 1
            }
        },
        {
            "type": "audio_url",
            "audio_url": {
                "url": "https://example.com/audio.mp3"
            }
        },
        {
            "type": "text",
            "text": "Compare and analyze the video visuals and audio content to find connections between them"
        }
    ]
}

5. Python SDK Usage Examples

5.1 Visual Recognition

from openai import OpenAI

client = OpenAI(
    api_key="your-api-key",
    base_url="https://api.siliconflow.cn/v1"
)

response = client.chat.completions.create(
    model="Qwen/Qwen2.5-VL-72B-Instruct",
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "image_url",
                    "image_url": {
                        "url": "https://example.com/tech-conference.jpg"
                    }
                },
                {
                    "type": "text",
                    "text": "What kind of tech conference is shown in this image? Analyze the expressions and atmosphere of the attendees"
                }
            ]
        }
    ]
)

print(response.choices[0].message.content)

5.2 Video Analysis

response = client.chat.completions.create(
    model="Qwen/Qwen3.5-397B-A17B",
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "video_url",
                    "video_url": {
                        "url": "https://example.com/product-demo.mp4",
                        "detail": "high",
                        "max_frames": 16,
                        "fps": 1
                    }
                },
                {
                    "type": "text",
                    "text": "What core features are demonstrated in this product demo video? What might be the target user group?"
                }
            ]
        }
    ],
    stream=True
)

# Streaming output
for chunk in response:
    content = chunk.choices[0].delta.content
    if content:
        print(content, end='', flush=True)

5.3 Audio Understanding

response = client.chat.completions.create(
    model="Qwen/Qwen3.5-397B-A17B",
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "audio_url",
                    "audio_url": {
                        "url": "data:audio/wav;base64,UklGRnoGAABXQVZFZm10IBAAAAABAAEA..."
                    }
                },
                {
                    "type": "text",
                    "text": "What is the main content and emotional tone of this audio segment?"
                }
            ]
        }
    ]
)

6. Billing Information

6.1 Visual Input Billing

Different models convert visual content differently. The following table shows core rules and billing metrics:

Model Series	Size/Pixel Constraints	detail=low	detail=high	Token Calculation Method
Qwen Series	Min `56×56`, Max `3584×3584`; Rounded to multiples of `28`	Unified `448×448`, ≈`256 tokens`	Width and height rounded up to multiples of `28`, then cropped proportionally to range	`ceil(h/28) * ceil(w/28)`
DeepseekVL2 Series	Base block `384×384`; `1 ≤ h*w ≤ 9` `(h,w)` blocks	Unified `384×384`, `421 tokens`	Scaled to `(h384, w384)`, selecting combination with maximum effective pixels	`(hw + 1) 196 + (w + 1) * 14 + 1`
GLM Series	Min `28×28`; Rounded to multiples of `28`; If smaller than `112×112` or exceeds limit, compressed to range	Unified `448×448`, ≈`256 tokens`	Width and height rounded to nearest multiple of `28` and limited to `(12544, 4816894)` pixel range	`(h/28) * (w/28)`

Note:

h,w are the final pixel dimensions used for billing; token counts in the table are estimates for visual input, actual billing is based on final conversion results at request time.

6.2 Video Input Billing

Video content is converted to tokens based on extracted frames:

Final frame count = min(fps × video duration, max_frames)
Each frame is converted according to the corresponding visual model standards

6.3 Audio Input Billing

Audio content is converted to tokens for billing. For Qwen3 omni multimodal models, input audio corresponds to 13 tokens per second, e.g., 22.5s audio corresponds to 292 tokens.

7. Best Practices

7.1 Performance Optimization

Video Duration Control: Recommended within 30 seconds for optimal analysis results
Frame Selection: max_frames=8-16, fps=1-2 is usually sufficient
Image Size: Preprocess according to model recommended dimensions

7.2 Usage Recommendations

Step-by-step Analysis: Break complex tasks into multiple simple steps
Multimodal Combination: Fully utilize advantages of different media types
Error Handling: Check media file accessibility and format compatibility

7.3 Common Questions

Q: File size limits? A: Recommended to keep audio/video files at moderate sizes, very large files may affect performance

Q: How many media files can be processed simultaneously? A: Multiple media URLs can be included in the same request, but overall data volume should be controlled

Q: Frame extraction strategy? A: For long videos, set reasonable fps and max_frames parameters for optimal analysis results and cost balance

Multimodal input

On this page