Raw data, clear context.

[
[
[

]
]
]

Say your programme needs to count the characters in café. Four looks like the obvious answer. Ask Rust for the size in bytes and you get five. Decide which count the programme needs before choosing a method.

A String owns UTF-8 text and can grow. To add the final é to caf, the programme uses push on a mutable String.[1] A &str is a shared view of UTF-8 text. A string literal already has that form.[1]

The counts function takes a &str. It accepts a borrow of word in the first call and a literal in the second. Both calls use the same function, and main keeps ownership of word.

On a &str, len() gives the byte count. chars() yields Rust char values, one per Unicode scalar value, so chars().count() counts those values.[2]

Create a Cargo project, then open its text-counts directory for the rest of the lesson.

Shell
cargo new --vcs none text-counts
Output
$ cargo new --vcs none text-counts
Creating binary (application) `text-counts` package
note: see more `Cargo.toml` keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html
exit=0

Replace src/main.rs with this programme and run it.

src/main.rs
Rust
fn counts(text: &str) {
println!("{}: {} bytes, {} scalar values", text, text.len(), text.chars().count());
}
fn main() {
let mut word = String::from("caf");
word.push('é');
counts(&word);
counts("cafe\u{0301}");
}
Shell
cargo run --quiet
Output
$ cargo run --quiet
café: 5 bytes, 4 scalar values
café: 6 bytes, 5 scalar values
exit=0

The first line uses a single scalar value for é. In the second, \u{0301} adds a combining acute accent after e. That spelling uses one extra scalar value and one extra byte.

In either spelling, the accented letter forms one grapheme cluster under Unicode’s default rules. Grapheme clusters approximate what people perceive as single characters.[3]

For a storage limit in bytes, use len(). If a limit is supposed to match what someone sees as a character, count grapheme clusters instead.

Two Unicode forms of café, both stored as UTF-8. Both start with hex bytes 63 61 66. The precomposed é (U+00E9) adds C3 A9, for five bytes and four scalar values. An e plus combining acute accent (U+0065 and U+0301) adds 65 CC 81, for six bytes and five scalar values.
The final letter can be stored as one scalar value or as an e followed by a combining acute accent. The byte tiles show the exact UTF-8 sequences from the example.

Why a single index fails

Rust rejects a single integer index into a String, even when its text contains only ASCII.[1]

Try the failing version by temporarily replacing src/main.rs and running the check below.

src/main.rs
Rust
fn main() {
let word = String::from("café");
let first = 0;
println!("{}", word[first]);
}
Shell
cargo check --quiet
Output
$ cargo check --quiet
error[E0277]: the type `str` cannot be indexed by `{integer}`
--> src/main.rs:4:25
|
4 | println!("{}", word[first]);
| ^^^^^ string indices are ranges of `usize`
|
= help: the trait `SliceIndex<str>` is not implemented for `{integer}`
= note: you can use `.chars().nth()` or `.bytes().nth()`
for more information, see chapter 8 in The Book: <https://doc.rust-lang.org/book/ch08-02-strings.html#indexing-into-strings>
help: `usize` implements trait `SliceIndex<T>`
--> /rustc/58cf8f9c35b355199727627d07ae34a4e2846d67/library/core/src/slice/index.rs:179:0
|
= note: `SliceIndex<[T]>`
--> /rustc/58cf8f9c35b355199727627d07ae34a4e2846d67/library/core/src/bstr/traits.rs:197:0
|
= note: `SliceIndex<ByteStr>`
= note: required for `String` to implement `Index<{integer}>`
For more information about this error, try `rustc --explain E0277`.
error: could not compile `text-counts` (bin "text-counts") due to 1 previous error
exit=101

The compiler points to the index and reports E0277. To walk through the scalar values, use chars() as in the working programme.

Try one more input

Restore the working programme. Change the second input to "🙂", predict its two counts, then rerun the same command. The first line should stay the same.

On the test run with Rust 1.99.1, the new input gave four bytes and one scalar value.

The next lesson uses slices to borrow part of a collection.

Browse Rust, one step at a time

Sources

  1. The Rust Programming Language: Storing UTF-8 Encoded Text with Strings
  2. Rust standard library: str
  3. Unicode Standard Annex #29: Unicode Text Segmentation

Leave a Reply