Stream Iterators and Predicates: From Files to Filters in Functional Style

Here's something that should blow your mind a little: the same .filter().map().collect() chain you use on a Vec works on a file. Line by line, without loading the entire file into memory, with the same syntax, the same performance characteristics, and the same zero-cost abstractions. A 10GB log file and a 10-element vector are processed through identical code paths—the only difference is where the data comes from.

This isn't an accident. It's the logical conclusion of Rust's iterator design: if everything that produces a sequence of values implements Iterator, then everything is composable with everything. In-memory collections, file streams, network sockets, stdin, random number generators—they all produce items through next(), and they all participate in the same pipeline ecosystem. Your algorithms don't care where the data comes from. Your pipelines don't know whether they're processing a Vec or a file. And that decoupling is what makes Rust's iterator model not just convenient, but architecturally powerful.

At RantAI, where our systems process data from databases, files, APIs, and real-time streams, the uniform iterator model is what keeps our pipeline code clean, testable, and reusable across data sources. This article, drawn from Chapter 4, Section 4.5.3 of our guide "The Rust Programming Language," shows how Rust's iterator model extends to I/O streams, and how closures as predicates enable expressive, composable filtering logic.

File Processing as an Iterator Pipeline#

use std::fs::File;
use std::io::{BufRead, BufReader};

fn main() -> std::io::Result<()> {
    let file = File::open("server.log")?;
    let reader = BufReader::new(file);

    // Process a log file line by line — never loads the entire file
    let error_count = reader.lines()
        .filter_map(|line| line.ok())           // Skip lines that fail to read
        .filter(|line| line.contains("ERROR"))  // Keep only error lines
        .count();                                // Count them

    println!("Found {} error lines", error_count);
    Ok(())
}

BufReader::lines() returns an iterator that yields one line at a time, reading from the file on demand. The file is never loaded entirely into memory—each line is read, processed, and discarded before the next is read. This means you can process a 100GB file with a few kilobytes of memory, using the exact same syntax you'd use for a Vec<String>.

The .filter_map(|line| line.ok()) step deserves attention: lines() yields Result<String, io::Error> (because any line read might fail). filter_map combines filter and map: it applies a function that returns Option<T>, keeping Some values and discarding None. Since Result::ok() converts Ok(v) to Some(v) and Err(e) to None, this neatly skips failed reads without crashing.

Closures as Predicates: Parameterized Filtering#

A predicate is just a closure that returns bool. By storing predicates in variables, you can build reusable, composable filter logic:

fn main() {
    let data = vec![
        "2024-01-15 ERROR database connection failed",
        "2024-01-15 INFO server started",
        "2024-01-15 WARN disk usage high",
        "2024-01-15 ERROR timeout exceeded",
        "2024-01-16 INFO request processed",
        "2024-01-16 ERROR authentication failed",
    ];

    // Predicates as closures — reusable filter logic
    let is_error = |line: &&str| line.contains("ERROR");
    let is_recent = |line: &&str| line.starts_with("2024-01-16");

    // Compose predicates
    let errors: Vec<&str> = data.iter().filter(is_error).copied().collect();
    println!("All errors: {:?}", errors);

    let recent_errors: Vec<&str> = data.iter()
        .filter(is_error)
        .filter(is_recent)
        .copied()
        .collect();
    println!("Recent errors: {:?}", recent_errors);

    // Count without collecting
    let error_count = data.iter().filter(is_error).count();
    println!("Error count: {}", error_count);  // 3
}

Predicates stored as closures are reusable across multiple operations. The same is_error predicate works with filter, any, all, find, and count. This is the functional programming approach to filtering: define what to filter, then apply it wherever you need it.

Building Predicates Dynamically#

fn make_severity_filter(level: &str) -> impl Fn(&&str) -> bool + '_ {
    move |line: &&str| line.contains(level)
}

fn main() {
    let logs = vec![
        "ERROR: disk full",
        "WARN: memory high",
        "INFO: all systems go",
        "ERROR: network down",
    ];

    let filter_errors = make_severity_filter("ERROR");
    let errors: Vec<&&str> = logs.iter().filter(filter_errors).collect();
    println!("Errors: {:?}", errors);
}

Functions that return closures create parameterized predicates—filter factories that produce custom filters based on runtime configuration. This pattern is powerful for building configurable data processing pipelines.

Collecting into Sorted, Unique Sets: BTreeSet#

use std::collections::BTreeSet;
use std::fs::File;
use std::io::{BufRead, BufReader};

fn unique_words_from_file(path: &str) -> std::io::Result<BTreeSet<String>> {
    let file = File::open(path)?;
    let reader = BufReader::new(file);

    let words: BTreeSet<String> = reader.lines()
        .filter_map(|line| line.ok())
        .flat_map(|line| {
            line.split_whitespace()
                .map(|w| w.to_lowercase())
                .collect::<Vec<_>>()
        })
        .collect();

    Ok(words)
}

BTreeSet gives you sorted, unique elements—useful when you need both deduplication and ordering. Unlike HashSet (which is unordered), BTreeSet maintains elements in sorted order, making it perfect for building indexes, sorted word lists, or any output where order matters.

The flat_map deserves highlighting: each line produces multiple words, and flat_map flattens the sequence of word-sequences into a single sequence of words. It's the iterator equivalent of map + flatten—transforming each element into multiple elements and then concatenating the results.

Console Stream Processing#

use std::io::{self, BufRead};

fn main() {
    println!("Enter words (Ctrl+D to finish):");

    let unique_words: std::collections::BTreeSet<String> = io::stdin()
        .lock()
        .lines()
        .filter_map(|line| line.ok())
        .flat_map(|line| {
            line.split_whitespace()
                .map(|w| w.to_lowercase().to_string())
                .collect::<Vec<_>>()
        })
        .collect();

    println!("Unique words (sorted): {:?}", unique_words);
}

The exact same pipeline works on stdin. The data source changed from a file to a keyboard, but the processing logic is identical. This is the uniform iterator model in action: algorithms don't care where data comes from.

The Uniform Processing Model#

This is the key insight worth emphasizing: in Rust, the processing pipeline is independent of the data source. You write your transformations once—filter, map, collect—and they work with:

• Vec::iter() → in-memory data

• BufReader::lines() → file data

• stdin().lock().lines() → console input

• Custom iterators → any data source you define

At RantAI, we exploit this uniformity to build testable pipelines. Our production code processes files and network streams. Our tests feed the same pipeline with Vec data. The pipeline logic is identical—only the data source changes. This makes testing trivial and refactoring safe.

Broader Implications: Data Source as a Plugin#

When your algorithms are generic over Iterator, adding a new data source is just implementing next(). Adding a new algorithm is just writing a function that takes impl Iterator. The two dimensions—data sources and algorithms—grow independently. This is the kind of architectural decoupling that large systems need to remain maintainable as they grow.

Practical Applications & Strategic Takeaways#

For newcomers: Remember that BufReader::lines() is an iterator. Everything you've learned about filter, map, and collect works on files. Start processing files with iterator chains instead of manual line-reading loops.

For data engineers: flat_map is your friend for one-to-many transformations (one line → many words, one record → many fields). Combined with collect::<BTreeSet<_>>(), it gives you sorted, unique output from any data source.

For architects: Design your pipelines to accept impl Iterator<Item = T> rather than specific data sources. This makes your processing logic data-source-agnostic and trivially testable with in-memory data.

Our Commitment to Open Knowledge#

RantAI is committed to open education. Stream iterators and predicates are covered in Chapter 4, Section 4.5.3 of our guide, "The Rust Programming Language," freely available online.

Explore these concepts further: https://trpl.rantai.dev

Support Our Mission & Get Your Handbook#

• Get the Handbook on Amazon KDP: https://www.amazon.com/dp/B0DHCMD3F2

• Get the Handbook on Google Play Books: https://play.google.com/store/books/details?id=INwfEQAAQBAJ

What's the largest file you've processed with an iterator pipeline? Or the most satisfying moment when you realized the same pipeline worked on both files and in-memory data? Share your stream processing stories below!

#RustLang #Iterators #StreamProcessing #FileIO #Predicates #RantAI #LearnRust #FunctionalProgramming #DataPipelines #ZeroCost