Increase performance by reducing I/O
Rather than splitting content from CSV files, redirecting them into newly created temporary files and appending them on the target CSV file again (2 Reads, 2 Writes, 1 file creation (write), 1 file deletion (write)), we now pipe the output of the tail
call within system2()
directly into the target CSV file via tee
(1 Read, 1 Write). This greatly improves performance, particularly on slow disks (network shares, etc).
Further optimizations might be a lot harder. system2()
can generally work asynchronously system2(..., wait = FALSE)
, which however could potentially lead to race conditions where two processes write into the same line. More performance may only be achievable via data-base (style) storage.