Blog - Data Science and the fst Package

Lightning fast serialization of data frames using the fst package

Wed, 10 Jan 2018 00:00:00 +0000

Version 0.8.0 of R’s fst package has been released to CRAN. The package is now multi-threaded allowing for even faster serialization of data frames to disk.

Multi-threaded LZ4 and ZSTD compression from R

Wed, 10 Jan 2018 00:00:00 +0000

The fst package uses LZ4 and ZSTD to compress columnar data. In the latest release, methods compress_fst and decompress_fst were added which allow for direct (multi-threaded) access to these excellent compressors.

Multi-threaded hashing with xxHash from R

Wed, 10 Jan 2018 00:00:00 +0000

The fst package uses the xxHash algorithm for internal hashing of (meta-)data. With method hash_fst the user now has direct access to this extremely fast hashing algorithm.