To safely manipulate UTF-8 strings, you must use multibyte-aware functions because standard string operations assume one byte per character, which corrupts multibyte characters in UTF-8; 1. Always use Unicode-safe functions like mb_substr() and mb_strlen() in PHP with 'UTF-8' encoding specified; 2. In Python 3, strings are Unicode by default, but ensure files are opened with encoding='utf-8' and avoid byte-level slicing; 3. In JavaScript, use spread operator [...text] or Array.from() for accurate character counting and slicing due to surrogate pairs; 4. Always specify UTF-8 when reading/writing files, databases, or network data; 5. Avoid byte-level operations unless handling raw UTF-8 streams deliberately; 6. Validate input as valid UTF-8 before processing; 7. Test with real-world text including emojis, accents, and CJK characters; 8. When working with raw bytes, only split at valid UTF-8 boundaries by checking leading bits (0xxxxxxx for 1-byte, 110xxxxx for 2-byte, 1110xxxx for 3-byte, 11110xxx for 4-byte) and never split within a multibyte sequence; the safest approach is to decode to Unicode, modify, then re-encode, ensuring data integrity across global user inputs.

When working with text in modern web and software development, handling UTF-8 properly is non-negotiable. While ASCII covers basic English characters, UTF-8 supports virtually every language on the planet—using multibyte sequences to represent characters outside the 7-bit range. But this flexibility introduces complexity, especially when modifying strings: slicing, replacing, inserting, or reversing. Do it wrong, and you risk corrupting data, breaking user experience, or introducing subtle bugs.

Here’s what you need to know to safely manipulate UTF-8 strings.

Why Regular String Functions Fail with UTF-8

Many programming languages provide built-in string functions that assume one character equals one byte. That works fine for ASCII, but fails with UTF-8, where:

• Basic Latin letters (A–Z) use 1 byte
• Accented characters (like é, ?) often use 2 bytes
• Most Chinese, Japanese, and Korean characters use 3 or 4 bytes
• Emojis typically use 4 bytes

Example: The emoji "?" is 4 bytes in UTF-8 (F0 9F 98 8A in hex). If you try to slice it at byte 2, you’ll split the sequence mid-way—resulting in invalid or garbled output.

Using functions like substr() in PHP (without mb_ prefix), or len() and slicing in Python without considering Unicode, can lead to:

• Truncated characters (mojibake: "")
• Incorrect string length (counting bytes instead of characters)
• Broken text processing (e.g., capitalizing half a character)

Bottom line: You must use multibyte-aware functions when dealing with UTF-8.

Use Multibyte String Functions

Most modern languages offer Unicode-safe alternatives. Here's how to handle UTF-8 correctly in common environments.

PHP: Always Use mbstring

PHP’s default string functions are byte-based. Enable and use the mbstring extension:

// ? Wrong – breaks UTF-8
$truncated = substr("café", -1); // Might break if 'é' is 2 bytes

// ? Correct – respects UTF-8
$truncated = mb_substr("café", -1, 1, 'UTF-8'); // Returns "é"
$length    = mb_strlen("café", 'UTF-8');         // Returns 4

Make sure mbstring is enabled and consider setting:

mb_internal_encoding('UTF-8');

Python: Unicode Strings Are Default (in Python 3)

In Python 3, strings are Unicode by default, so high-level operations are generally safe:

text = "Hello ?"
print(len(text))          # 8 (correct, including emoji)
print(text[6:])           # "?" (correct slicing)

But be cautious with:

• Encoding/decoding: Always specify encoding when reading files:

  with open('file.txt', encoding='utf-8') as f:
      content = f.read()

• Byte operations: Avoid slicing bytes objects unless you’re handling raw UTF-8 data.

JavaScript: Mostly Safe, But Watch Edge Cases

JavaScript uses UTF-16 internally, not UTF-8—but handles most Unicode well:

const text = "Hello ?";
console.log(text.length); // 7 (but ? is a surrogate pair = 2 units)

The catch: some characters (like many emojis) are represented as surrogate pairs (2 code units), so .length gives 2, not 1.

For accurate character count and safe slicing, use:

[...text].length;         // 6 (correct number of characters)
[...text].slice(-1)[0];   // "?"

Or use Array.from() which handles surrogates and combining marks properly.

Key Rules for Safe UTF-8 String Manipulation

To avoid corruption when modifying multibyte strings:

• ? Always specify encoding when reading/writing files, databases, or network data.
• ? Use multibyte functions (mb_* in PHP, Unicode-aware methods elsewhere).
• ? Avoid byte-level operations unless you’re parsing UTF-8 streams deliberately.
• ? Validate input—ensure text is valid UTF-8 before processing.
• ? Test with real-world text: Use strings with emojis, accents, and CJK characters.

One common pitfall: trimming or truncating user input for display without respecting character boundaries. A 10-character limit should mean 10 characters, not 10 bytes.

When You Must Work with Bytes

Sometimes you're dealing with raw UTF-8 byte streams (e.g., network protocols, file parsing). In those cases:

1. Don’t split in the middle of a multibyte sequence

   • UTF-8 uses leading bits to indicate byte length:
     • 0xxxxxxx → 1 byte (ASCII)
     • 110xxxxx → start of 2-byte sequence
     • 1110xxxx → start of 3-byte
     • 11110xxx → start of 4-byte
   • Continuation bytes are 10xxxxxx

2. Only split at valid boundaries:

   • After a 0xxxxxxx byte
   • After a complete multibyte sequence (e.g., after a 4-byte emoji)

You can write a helper to find safe split points, but it’s easier to decode to Unicode first, then re-encode after modification.

Handling UTF-8 safely isn’t hard once you respect its structure. The key is using the right tools and never assuming one byte equals one character. Whether you're trimming a username or parsing a multilingual document, treat UTF-8 with care—your global users will thank you.

Basically: use Unicode-aware functions, test with real text, and never trust byte-based slicing.

The above is the detailed content of Handling UTF-8: A Deep Dive into Multibyte String Modification. For more information, please follow other related articles on the PHP Chinese website!