process_files_parallel

Function process_files_parallel 

Source
pub fn process_files_parallel<T, F, C, R>(
    files: &[String],
    processor: F,
    combiner: C,
) -> Result<(R, usize)>
where T: Send + 'static, R: Send + 'static, F: Fn(&str) -> Result<T> + Send + Sync, C: Fn(Vec<FileProcessResult<T>>) -> R + Send + Sync,
Expand description

Process multiple files in parallel with memory-efficient streaming

This function processes files concurrently using a streaming approach that processes entries line-by-line without loading entire files into memory. This prevents memory allocation failures on large files.

§Arguments

  • files - List of file paths to process
  • processor - Function that processes a single file with streaming callback
  • combiner - Function that combines results from all files

§Returns

Combined result from all files plus total lines processed