Always slice strings by characters, not bytes, to avoid corrupting multi-byte UTF-8 sequences. 1. Understand that UTF-8 characters can be 1–4 bytes, so byte-based slicing can split characters. 2. Avoid treating strings as byte arrays; use decoded Unicode strings for slicing. 3. Decode bytes to text early and encode only when necessary. 4. Use Unicode-aware libraries like Python’s regex or JavaScript’s Intl.Segmenter for grapheme-safe slicing. 5. Validate or repair partial byte sequences if working with raw bytes. Incorrect slicing causes garbled text, especially with emojis or international characters, so always ensure slices align with character boundaries.

When working with strings in modern programming, especially in web applications or systems handling international text, it’s easy to overlook how string slicing can silently corrupt data—especially when dealing with multi-byte characters like those in UTF-8. Incorrectly slicing such strings can break characters mid-sequence, resulting in garbled text, invalid encodings, or subtle bugs that are hard to trace. Here’s what you need to know to avoid these pitfalls.

? Understand That Not All Characters Are One Byte

The root of the problem lies in character encoding. In ASCII, each character takes exactly one byte, so slicing by byte index aligns perfectly with character boundaries. But in UTF-8 (the most common encoding today), characters can take 1 to 4 bytes:

• ASCII letters (a-z): 1 byte
• Accented characters (é, ?): 2–3 bytes
• Emojis (?, ?): 4 bytes

If you slice a UTF-8 string at a byte index that falls inside a multi-byte character, you’ll split that character and produce invalid or corrupted output.

Example:

text = "Hello ?"
# UTF-8 bytes: b'Hello \xf0\x9f\x98\x82'
# If you slice at byte 7: b'Hello \xf0' → invalid

Slicing at byte 7 cuts into the 4-byte ? emoji, leaving an incomplete byte sequence. When decoded, this may result in a replacement character () or cause a decoding error.

? Common Pitfall: Treating Strings as Byte Arrays

Many languages allow you to slice strings by index, but the behavior depends on whether the index refers to code units, code points, or grapheme clusters.

Problematic Approaches:

• Using byte indices in Python without care:

  # WRONG: This slices by Unicode code points, but truncates visually
  s = "café?"
  print(s[:4])  # 'café' – okay
  print(s[:5])  # 'café?' – good

  But if you're working with raw bytes:

  s_bytes = "café?".encode('utf-8')
  truncated = s_bytes[:6]  # Truncate to 6 bytes
  print(truncated.decode('utf-8', errors='replace'))  # 'caf' – corrupted!

  The 'é' is 2 bytes in UTF-8 (\xc3\xa9), so slicing at byte 6 may cut it in half.

• JavaScript substring on emoji:

  "?".substring(0, 1); // Returns "\ud83d" – high surrogate only → invalid

  JavaScript uses UTF-16, where emojis are two "surrogate" code units. Slicing between them creates broken characters.

? Safe Practices for Slicing Strings

To avoid corruption, always slice strings at valid character (code point or grapheme) boundaries, not arbitrary byte or code unit positions.

1. Use Language Features That Respect Unicode

In Python, use the string itself (not raw bytes) for slicing:

s = "Hello ?!"
safe_slice = s[:6]  # 'Hello ' – includes space, avoids breaking emoji

But even better: ensure you’re not mixing byte and text operations.

2. Decode Early, Encode Late

Work with decoded strings (Unicode) in memory, and only encode to bytes when outputting:

# GOOD
data = get_bytes_from_network()
text = data.decode('utf-8')  # Full decode first
safe_part = text[:10]        # Safe slicing
result = safe_part.encode('utf-8')  # Encode only when needed

3. Use Libraries for Grapheme-Aware Slicing

Some characters are grapheme clusters — visually single characters made of multiple code points (e.g., "é" as e ′, or flags like ??).

Use libraries like:

• Python: unicodedata, or regex module (supports \X for graphemes)
• JavaScript: Intl.Segmenter, or libraries like grapheme-splitter

Example in Python with regex:

import regex

text = "Hello ??!"
# Split into graphemes
chunks = regex.findall(r'\X', text)
safe_slice = ''.join(chunks[:6])  # Safely includes whole emoji/flags

4. Validate After Slicing

If you must work with byte slices (e.g., streaming data), validate UTF-8 integrity:

def safe_decode(b: bytes) -> str:
    try:
        return b.decode('utf-8')
    except UnicodeDecodeError:
        # Try to find the last valid start of a UTF-8 sequence
        for i in range(len(b) - 1, -1, -1):
            try:
                tail = b[i:].decode('utf-8')
                head = b[:i].decode('utf-8', errors='ignore')
                return head   tail
            except UnicodeDecodeError:
                continue
        return b.decode('utf-8', errors='replace')

Key Takeaways

• Never assume one character = one byte.
• Avoid slicing raw UTF-8 bytes unless you’re handling boundaries correctly.
• Decode to Unicode strings early and slice in the string domain.
• For UI/text display, consider grapheme clusters, not just code points.
• Validate or repair UTF-8 when dealing with partial byte sequences.

Corrupting strings via incorrect slicing might seem rare, but it shows up in truncating log messages, generating previews, or processing user input—especially from global users. Handle text with care, and treat UTF-8 boundaries with respect.

Basically: slice characters, not bytes.

The above is the detailed content of Avoiding Corrupted Data: Pitfalls of Slicing Multi-byte Strings Incorrectly. For more information, please follow other related articles on the PHP Chinese website!