046: Character Encoding and UTF-8 in PHP

What are character encodings?

• Comparison of number systems (base-10, binary, hexadecimal)
• Historical development of character maps and encodings (telegram, ASCII, ISO-8859-1...)
• Unicode and its implementations (UTF-8, UTF-16, UTF-32)
  • UTF-8 uses 1-4 bytes per character--shows all Unicode characters while keeping a small file size
  • UTF-16 uses 2 bytes per character, so results in larger files
  • UTF-32 uses 4 bytes per character, so results in even larger files

Character encodings in PHP

• Some other languages have a UTF-8 core, while PHP stores strings a series of bytes--can result in misinterpretation/mojibake
• Mind the multibyte string functions:
  • E.g., strpos() works by counting a number of bytes; this is unreliable for UTF-8, so use mb_strpos() for unicode strings
  • There is a php.ini setting which overloads functions like strpos with their multibyte versions. This it NOT RECOMMENDED as it can cause weird bugs, whether when switching hosts or when using 3rd-party code that wasn't written with this in mind.
• php7: the IntlChar class offers many useful static functions, e.g. IntlChar::chr() and IntlChar::isWhitespace
• php7: \u escape for unicode code points in strings

Character encodings in HTML

• HTML5: <meta charset="utf-8">
• Content-Type: text/html; charset=utf-8 HTTP header
• <form enctype="text/plain; charset=utf-8">

Character encodings in MySQL

• MySQL default encoding is still latin1_swedish_ci--beware
• Only utf8mb4 supports the full range of UTF-8 characters (as people have discovered from trying to store emoji)
• varbinary and blob consume less space than utf8 varchar, so they are useful in fields that users won't touch or whose contents never need to include special chars (e.g. URLs)
  • Is there any other tradeoff?

Tips/best practices

• Save source files as UTF-8 without the Byte Order Mark (BOM).
• Always use <meta> and a Content-Type header.
  • Remember that header() must precede any echoed output. (Presence of a BOM can cause bugs here.)
• Storing UTF-8 special characters in their native form (rather than as escaped sequences) in your source can act as "canary" for others--if I see mojibake, maybe my IDE/editor is not configured correctly?
• For MySQL, use SET NAMES utf8 at the beginning of every connection
• Always validate inputs; always consider character encoding for input (whether from user or APIs), persistence, and output.
• Remember that connections/clients themselves also have encodings
• Mismatched encoding bugs can hide among Roman alphanumerics, since ISO-8859 and UTF-8 are compatible in these lower code points. Check higher code points to be sure.

Other resources

• The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!)
• Data Encoding: A Guide to UTF-8 for PHP and MySQL