Bytes, text and JSON

A byte is a small number, anything from 0 to 255. Every file, message and picture a computer handles is, underneath, a long row of bytes.

Text is letters, digits and symbols, the things people read. To send text, a computer needs an agreed rule that turns each character into bytes and back again. That rule is called an encoding. A byte on its own means nothing. The byte 65 is the letter A only because both sides agreed to read it as text.

The oldest common rule is ASCII. It covers English letters, the digits, punctuation and a space, and gives each one a single byte. Capital A is 65. The digit 0 is 48.

ASCII has no room for é, the degree sign, Chinese or emoji. UTF-8 is the rule that covers all of them. It uses one byte for every ASCII character, exactly as ASCII does. Every other character takes two, three or four bytes. The degree sign takes two. A seedling emoji takes four.

So here is the trap. Counting the characters you see does not tell you how many bytes there are. "21.5 °C" looks like 7 characters, but it is 8 bytes. Memory and message limits are counted in bytes, so a program must count bytes.

This section builds three ideas: bytes, text, and JSON, a text format for structured data that the last part of the section teaches. A byte is 8 bits, so it holds one of 256 values, 0 to 255, written in hex as 0x00 to 0xFF. Text is a sequence of characters. An encoding is the rule that turns each character into bytes and back. Bytes have no meaning of their own: 0x41 is the letter A under one encoding, the number 65 to a program reading an unsigned byte, and part of a larger number to a program reading two bytes at once.

ASCII is the oldest encoding still in daily use. It assigns 128 characters, English letters, digits, punctuation and a few control codes, to the values 0–127, one byte each. A is 0x41, a is 0x61, the digit 0 is 0x30, a space is 0x20, { is 0x7B.

UTF-8 encodes every character in Unicode, the catalogue of more than 150 000 characters from every writing system plus symbols and emoji. It uses 1 to 4 bytes per character, and its first 128 one-byte codes are exactly ASCII, so plain ASCII text is already valid UTF-8. Other characters take more:

So a character count is not a byte count. 21.5 °C is 7 characters and 8 bytes. Buffers, payload limits and radio packets are sized in bytes, so firmware must count the encoded bytes, not the characters a person sees. UTF-8 is the encoding of the web, of JSON, and of MQTT's topic names; assume it unless a document says otherwise.

Sometimes you need to see the actual bytes a program sent, not the text it was meant to be. A hex dump shows them.

It prints the bytes in rows of 16. Each byte appears as two hex digits. Hex is counting in sixteens, using the digits 0 to 9 and then the letters a to f. So 7b is 7 sixteens plus 11, which is 123.

Each row starts with a position number, also in hex, called the offset. It says how far into the data that row begins, counting from 0. The second row starts at 00000010, which is 16 in ordinary numbers. To find any byte's position, add its row's offset to its place along the row.

On the right, the same bytes are shown as characters, where that makes sense. Anything that isn't a plain letter, digit or symbol shows as a dot. That's how you spot a hidden line break at the end of a message, or a degree sign that arrived broken.

A hex dump shows raw bytes as text, one row per 16 bytes. Each row starts with an offset, the position of its first byte counted from 0 and written in hex, then the 16 bytes as two hex digits each, then the same bytes as characters, with a dot for any byte that isn't printable ASCII. Here are the 18 bytes of the JSON text {"t":21.5,"rh":64}, as hexdump -C or xxd-style tools print them:

00000000  7b 22 74 22 3a 32 31 2e  35 2c 22 72 68 22 3a 36  |{"t":21.5,"rh":6|
00000010  34 7d                                             |4}|

The second row's offset is 0x10, which is 16: it starts with the seventeenth byte. A byte's offset is its row's offset plus its column, so 7d (the closing brace) is at \texttt{0x10} + 1 = \texttt{0x11}, or 17. Reading a hex dump answers questions a printed string hides: whether a payload ends with an invisible newline (0a), whether a degree sign arrived as the two bytes c2 b0 or got mangled, whether a number was sent as text digits (32 31) or as a binary value.

JSON is a way of writing data as text so that any program can read it. It's short for JavaScript Object Notation, but every language uses it now.

It has a handful of building blocks. Words and names go in double quotes, like "bench1". Numbers are written plainly, with no quotes and no units, like 21.5. There are also true, false and null, which means "no value."

Curly braces make an object: a set of labels, each with a value. {"t": 21.5, "rh": 64} has two labels. t holds 21.5, the temperature, and rh holds 64, the relative humidity. A colon goes between a label and its value, and commas go between pairs.

Square brackets make an array: a list in order, like [21.5, 21.6, 21.4]. Objects and arrays can sit inside each other.

Why do devices use JSON so much? The labels travel with the numbers, so the receiver knows which number is which. Every programming language can read it. And people can read it too, in a log or a hex dump.

The price is size. Those two readings take 18 bytes as JSON. If both sides agreed on a fixed layout in advance, they could fit in 3 bytes.

JSON (JavaScript Object Notation) is a text format for structured data. A JSON document is one value, and a value is one of:

• a string in double quotes: "bench1";
• a number, with no quotes and no units: 21.5, 64, -3;
• true, false or null;
• an object: { key-value pairs }, where each key is a string, a colon separates it from its value, and commas separate pairs: {"t": 21.5, "rh": 64};
• an array: an ordered list of values in brackets: [21.5, 21.6, 21.4].

Objects and arrays nest, so {"bench": "b1", "temps": [21.5, 21.6]} is one object holding a string and an array. JSON text is UTF-8, keys are case-sensitive, and whitespace outside strings means nothing, so {"t":21.5} and { "t": 21.5 } are the same document in 10 and 13 bytes.

JSON is the most common payload in this course and in device systems generally, for three reasons. It is self-describing: the keys travel with the values, so a reader can tell t from rh without a separate format description. Every language this course uses parses it with its standard library or one well-known package (Python's json, ArduinoJson on the board). And it is readable in a hex dump and in a log.

Its cost is size. The two values above take 18 bytes as compact JSON. A binary encoding that both sides agree on in advance could send the temperature as a 16-bit integer in hundredths of a degree (2150) and the humidity as one byte: 3 bytes. CBOR (Concise Binary Object Representation) is a standard binary format with JSON's data model that sits between the two; Chapter 10 §6 compares them properly.