Reading and writing ffdf data.frame using ff subscripts

Function ffdfindexget allows to extract rows from an ffdf data.frame according to positive integer suscripts stored in an ff vector.
Function ffdfindexset allows the inverse operation: assigning to rows of an ffdf data.frame according to positive integer suscripts stored in an ff vector. These functions allow more control than the method dispatch of [ and [<- if an ff integer subscript is used.

Usage

ffdfindexget(x, index, indexorder = NULL, autoindexorder = 3, FF_RETURN = NULL
  , BATCHSIZE = NULL, BATCHBYTES = getOption("ffmaxbytes"), VERBOSE = FALSE)
  ffdfindexset(x, index, value, indexorder = NULL, autoindexorder = 3
  , BATCHSIZE = NULL, BATCHBYTES = getOption("ffmaxbytes"), VERBOSE = FALSE)

Arguments

x: A ffdf data.frame containing the elements
index: A ff integer vector with integer subscripts in the range from 1 to length(x).
value: A ffdf data.frame like x with the rows to be assigned
indexorder: Optionally the return value of ffindexorder, see details
autoindexorder: The minimum number of columns (which need chunked indexordering) for which we switch from on-the-fly ordering to stored ffindexorder
FF_RETURN: Optionally an ffdf data.frame of the same type as x in which the returned values shall be stored, see details.
BATCHSIZE: Optinal limit for the batchsize (see details)
BATCHBYTES: Limit for the number of bytes per batch
VERBOSE: Logical scalar for verbosing

Details

Accessing rows of an ffdf data.frame identified by integer positions in an ff vector is a non-trivial task, because it could easily lead to random-access to disk files. We avoid random access by loading batches of the subscript values into RAM, order them ascending, and only then access the ff values on disk. Such ordering is don on-thy-fly for upto autoindexorder-1 columns that need ordering. For autoindexorder o more columns we do the batched ordering upfront with ffindexorder and then re-use it in each call to ffindexget resp. ffindexset.

Value

Function ffdfindexget returns a ffdf data.frame with those rows selected by the ff index vector.
Function ffdfindexset returns x with those rows replaced that had been requested by index and value.

Author

Jens Oehlschlägel

Examples

message("ff integer subscripts with ffdf return/assign values")
#> ff integer subscripts with ffdf return/assign values
x <- ff(factor(letters))
y <- ff(1:26)
d <- ffdf(x,y)
i <- ff(2:9)
di <- d[i,]
di
#> ffdf (all open) dim=c(8,2), dimorder=c(1,2) row.names=NULL
#> ffdf virtual mapping
#>   PhysicalName VirtualVmode PhysicalVmode  AsIs VirtualIsMatrix
#> x            x      integer       integer FALSE           FALSE
#> y            y      integer       integer FALSE           FALSE
#>   PhysicalIsMatrix PhysicalElementNo PhysicalFirstCol PhysicalLastCol
#> x            FALSE                 1                1               1
#> y            FALSE                 2                1               1
#>   PhysicalIsOpen
#> x           TRUE
#> y           TRUE
#> ffdf data
#>   x y
#> 1 b 2
#> 2 c 3
#> 3 d 4
#> 4 e 5
#> 5 f 6
#> 6 g 7
#> 7 h 8
#> 8 i 9
d[i,] <- di
message("ff integer subscripts: more control with ffindexget/ffindexset")
#> ff integer subscripts: more control with ffindexget/ffindexset
di <- ffdfindexget(d, i, FF_RETURN=di)
d <- ffdfindexset(d, i, di)
rm(x, y, d, i, di)
gc()
#>           used (Mb) gc trigger  (Mb) max used  (Mb)
#> Ncells 1242527 66.4    2239654 119.7  2239654 119.7
#> Vcells 2283920 17.5    8388608  64.0  5871749  44.8