Skip to content

Increase performance by reducing I/O

Weymeirsch, Jan requested to merge weymeirsch-main-patch-2548 into main

Rather than splitting content from CSV files, redirecting them into newly created temporary files and appending them on the target CSV file again (2 Reads, 2 Writes, 1 file creation (write), 1 file deletion (write)), we now pipe the output of the tail call within system2() directly into the target CSV file via tee (1 Read, 1 Write). This greatly improves performance, particularly on slow disks (network shares, etc).

Further optimizations might be a lot harder. system2() can generally work asynchronously system2(..., wait = FALSE), which however could potentially lead to race conditions where two processes write into the same line. More performance may only be achievable via data-base (style) storage.

Merge request reports

Loading