UTF-16

Yes.

The XML file must be saved using strict UTF-8 or UTF-16 Unicode encoding.

If a legacy domain controller generates the report with incorrect ANSI encoding, the Intune parser throws a fatal error.

And the most shocking limitation for multinational organizations.

That's how that number becomes actual bytes on disk.

UTF-8, UTF-16, Latin 1, Windows 252, all different answers to the same question.

Third, the grapheme.

That's what a human being looks at and calls one character.

Hold that thought because that's a whole separate failure pattern and it's the one Daniel ran into.

Then let's get the other structural facts on the table because there are a few and they don't blur together.

UTF-16.

UTF-16 is what Windows, Java, and JavaScript use internally.

You don't have to carry about like... bytes under there and byte encodings and whether or not they're null terminated or not.

And whether or not the like things are UTF aid or just like, I just want simple strings.

I know this is probably is a more privileged way to look at it because I don't have to deal with a bunch of other character sets very often, unless you include like emojis, which get weirdly complicated as well.

But like, I just want strings to be strings.

That would be really cool.

I'm sure a lot of people would hate it, but I would love it.

So there is such an important and useful use case for that, which is when you're compiling Rust to Wasm and you're interacting with JavaScript, and all the strings in JavaScript are UTF-16, or sometimes UCS2, which is even worse, but anyway, don't think about that, then if all your strings in Rust are UTF-8, which is what they are, then every time you interact with something that's in the browser DOM or something like that, there's a conversion happening.

So yeah, let's have a program-wide thing that just says all the strings in quotes are UTF-16.

So the name is...

UTF-8 smuggling, UTF-16 smuggling, actually, in most cases.

Yeah.

yes but yeah that's i mean i i shocked dan with that one in the past when i showed him like utf-8 and like unicode my favorite is when you do split javascript split on uh on emoji and then you turn for example the family into four emoji and these kind of things it's just yeah it's it's a mixture of text and binary that is very confusing for people