ffapply.rdThe ffapply functions support convenient batched processing of ff objects
such that each single batch or chunk will not exhaust RAM
and such that batchs have sizes as similar as possible, see bbatch.
Differing from R's standard apply which applies a FUNction,
the ffapply functions do apply an EXPRession and provide two indices FROM="i1" and TO="i2",
which mark beginning and end of the batch and can be used in the applied expression.
Note that the ffapply functions change the two indices in their parent frame, to avoid conflicts you can use different names through FROM="i1" and TO="i2".
For support of creating return values see details.
ffvecapply(EXPR, X = NULL, N = NULL, VMODE = NULL, VBYTES = NULL, RETURN = FALSE
, CFUN = NULL, USE.NAMES = TRUE, FF_RETURN = TRUE, BREAK = ".break"
, FROM = "i1", TO = "i2"
, BATCHSIZE = .Machine$integer.max, BATCHBYTES = getOption("ffbatchbytes")
, VERBOSE = FALSE)
ffrowapply(EXPR, X = NULL, N = NULL, NCOL = NULL, VMODE = NULL, VBYTES = NULL
, RETURN = FALSE, RETCOL = NCOL, CFUN = NULL, USE.NAMES = TRUE, FF_RETURN = TRUE
, FROM = "i1", TO = "i2"
, BATCHSIZE = .Machine$integer.max, BATCHBYTES = getOption("ffbatchbytes")
, VERBOSE = FALSE)
ffcolapply(EXPR, X = NULL, N = NULL, NROW = NULL, VMODE = NULL, VBYTES = NULL
, RETURN = FALSE, RETROW = NROW, CFUN = NULL, USE.NAMES = TRUE, FF_RETURN = TRUE
, FROM = "i1", TO = "i2"
, BATCHSIZE = .Machine$integer.max, BATCHBYTES = getOption("ffbatchbytes")
, VERBOSE = FALSE)
ffapply(EXPR = NULL, AFUN = NULL, MARGIN = NULL, X = NULL, N = NULL, DIM = NULL
, VMODE = NULL, VBYTES = NULL, RETURN = FALSE, CFUN = NULL, USE.NAMES = TRUE
, FF_RETURN = TRUE, IDIM = "idim"
, FROM = "i1", TO = "i2", BREAK = ".break"
, BATCHSIZE = .Machine$integer.max, BATCHBYTES = getOption("ffbatchbytes")
, VERBOSE = FALSE)the expression to be applied
ffapply only: alternatively to EXPR the name of a function to be applied, automatically converted to EXPR
ffapply only: the margins along which to loop in ffapply
an ff object from which several parameters can be derived, if they are not given directly: N, NCOL, NROW, DIM, VMODE, VBYTES, FF_RETURN
the total number of elements in the loop, e.g. number of elements in ffvecapply or number of rows in ffrowapply
ffrowapply only: the number of columns needed to calculate batch sizes
ffcolapply only: the number of rows needed to calculate batch sizes
ffapply only: the dimension of the array needed to calculate batch sizes
the vmode needed to prepare the RETURN object and to derive VBYTES if they are not given directly
the bytes per cell – see .rambytes – to calculate the RAM requirements per cell
the max number of bytes per batch, default getOption("ffbatchbytes")
an additional restriction on the number of loop elements, default=.Machine$integer.max
the name of the index that marks the beginning of the batch, default 'i1', change if needed to avoid naming-conflicts in the calling frame
the name of the index that marks the end of the batch, default 'i2', change if needed to avoid naming-conflicts in the calling frame
ffapply only: the name of an R variable used for loop-switching, change if needed to avoid naming-conflicts in the calling frame
ffapply only: the name of an R object in the calling frame that triggers break out of the batch loop, if 1) it exists 2) is.logical and 3) is TRUE
TRUE to prepare a return value (default FALSE)
name of a collapsing function, see CFUN
NULL gives return vector[1:N], RETCOL gives return matrix[1:N, 1:RETCOL]
NULL gives return vector[1:N], RETROW gives return matrix[1:RETROW, 1:N]
FALSE to return a ram object, TRUE to return an ff object, or an ff object that is ffsuitable to absorb the return data
FALSE to suppress attaching names or dimnames to the result
TRUE to verbose the batches
ffvecapply is the simplest ffapply method for ff_vectors. ffrowapply and ffcolapply is for ff_matrix,
and ffapply is the most general method for ff_arrays and ff_vectors.
There are many ways to change the return value of the ffapply functions.
In its simplest usage – batched looping over an expression – they don't return anything, see invisible.
If you switch RETURN=TRUE in ffvecapply then it is assumed that all looped expressions together return one vector of length N,
and via parameter FF_RETURN, you can decide whether this vector is in ram or is an ff object (or even which ff object to use).
ffrowapply and ffcolapply additionally have parameter RETCOL resp. RETROW which defaults to returning a matrix of the original size;
in order to just return a vector of length N set this to NULL, or specify a number of columns/rows for the return matrix.
It is assumed that the expression will return appropriate pieces for this return structure (see examples).
If you specify RETURN=TRUE and a collapsing function name CFUN, then it is assumed that the batched expressions return aggregated information,
which is first collected in a list, and finally the collapsing function is called on this list: do.call(CFUN, list). If you want to return the unmodified list,
you have to specify CFUN="list" for obvious reasons.
ffapply allows usages not completly unlike apply: you can specify the name of a function AFUN to be applied over MARGIN.
However note that you must specify RETURN=TRUE in order to get a return value.
Also note that currently ffapply assumes that your expression returns exactly one value per cell in DIM[MARGINS].
If you want to return something more complicated, you MUST specify a CFUN="list" and your return value will be a list with dim attribute DIM[MARGINS].
This means that for a function AFUN returning a scalar, ffapply behaves very similar to apply, see examples.
Note also that ffapply might create a object named '.ffapply.dimexhausted' in its parent frame,
and it uses a variable in the parent frame for loop-switching between dimensions, the default name 'idim' can be changed using the IDIM parameter.
Finally you can break out of the implied loops by assigning TRUE to a variable with the name in BREAK.
see details
xx The complete generation of the return value is preliminary and the arguments related to defining the return value might still change, especially ffapply is work in progress
message("ffvecapply examples")
#> ffvecapply examples
x <- ff(vmode="integer", length=100)
message("loop evaluate expression without returning anything")
#> loop evaluate expression without returning anything
ffvecapply(x[i1:i2] <- i1:i2, X=x, VERBOSE=TRUE)
#> ffvecapply 1:100{100} .... in 0 seconds
#> TOTAL TIME
#> user system elapsed
#> 0 0 0
ffvecapply(x[i1:i2] <- i1:i2, X=x, BATCHSIZE=20, VERBOSE=TRUE)
#> ffvecapply 1:20{100} .... in 0 seconds
#> ffvecapply 21:40{100} .... in 0 seconds
#> ffvecapply 41:60{100} .... in 0.001 seconds
#> ffvecapply 61:80{100} .... in 0 seconds
#> ffvecapply 81:100{100} .... in 0 seconds
#> TOTAL TIME
#> user system elapsed
#> 0.001 0.000 0.001
ffvecapply(x[i1:i2] <- i1:i2, X=x, BATCHSIZE=19, VERBOSE=TRUE)
#> ffvecapply 1:17{100} .... in 0 seconds
#> ffvecapply 18:34{100} .... in 0.001 seconds
#> ffvecapply 35:51{100} .... in 0 seconds
#> ffvecapply 52:68{100} .... in 0 seconds
#> ffvecapply 69:85{100} .... in 0 seconds
#> ffvecapply 86:100{100} .... in 0 seconds
#> TOTAL TIME
#> user system elapsed
#> 0.001 0.000 0.001
message("lets return the combined expressions as a new ff object")
#> lets return the combined expressions as a new ff object
ffvecapply(i1:i2, N=length(x), VMODE="integer", RETURN=TRUE, BATCHSIZE=20)
#> ff (open) integer length=100 (100)
#> [1] [2] [3] [4] [5] [6] [7] [8] [93] [94] [95] [96]
#> 1 2 3 4 5 6 7 8 : 93 94 95 96
#> [97] [98] [99] [100]
#> 97 98 99 100
message("lets return the combined expressions as a new ram object")
#> lets return the combined expressions as a new ram object
ffvecapply(i1:i2, N=length(x), VMODE="integer", RETURN=TRUE, FF_RETURN=FALSE, BATCHSIZE=20)
#> [1] 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
#> [19] 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36
#> [37] 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54
#> [55] 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72
#> [73] 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90
#> [91] 91 92 93 94 95 96 97 98 99 100
#> attr(,"physical")
#> (hidden, use physical(x) to access the physical attributes and vmode(x) for accessing vmode)
#> attr(,"virtual")
#> (hidden, use virtual(x) to access the virtual attributes)
#> attr(,"vmode")
#> [1] "integer"
message("lets return the combined expressions in existing ff object x")
#> lets return the combined expressions in existing ff object x
x[] <- 0L
ffvecapply(i1:i2, N=length(x), VMODE="integer", RETURN=TRUE, FF_RETURN=x, BATCHSIZE=20)
#> ff (open) integer length=100 (100)
#> [1] [2] [3] [4] [5] [6] [7] [8] [93] [94] [95] [96]
#> 1 2 3 4 5 6 7 8 : 93 94 95 96
#> [97] [98] [99] [100]
#> 97 98 99 100
x
#> ff (open) integer length=100 (100)
#> [1] [2] [3] [4] [5] [6] [7] [8] [93] [94] [95] [96]
#> 1 2 3 4 5 6 7 8 : 93 94 95 96
#> [97] [98] [99] [100]
#> 97 98 99 100
message("aggregate and collapse")
#> aggregate and collapse
ffvecapply(summary(x[i1:i2]), X=x, RETURN=TRUE, CFUN="list", BATCHSIZE=20)
#> $`1:20`
#> Min. 1st Qu. Median Mean 3rd Qu. Max.
#> 1.00 5.75 10.50 10.50 15.25 20.00
#>
#> $`21:40`
#> Min. 1st Qu. Median Mean 3rd Qu. Max.
#> 21.00 25.75 30.50 30.50 35.25 40.00
#>
#> $`41:60`
#> Min. 1st Qu. Median Mean 3rd Qu. Max.
#> 41.00 45.75 50.50 50.50 55.25 60.00
#>
#> $`61:80`
#> Min. 1st Qu. Median Mean 3rd Qu. Max.
#> 61.00 65.75 70.50 70.50 75.25 80.00
#>
#> $`81:100`
#> Min. 1st Qu. Median Mean 3rd Qu. Max.
#> 81.00 85.75 90.50 90.50 95.25 100.00
#>
ffvecapply(summary(x[i1:i2]), X=x, RETURN=TRUE, CFUN="crbind", BATCHSIZE=20)
#> Min. 1st Qu. Median Mean 3rd Qu. Max.
#> 1:20 1 5.75 10.5 10.5 15.25 20
#> 21:40 21 25.75 30.5 30.5 35.25 40
#> 41:60 41 45.75 50.5 50.5 55.25 60
#> 61:80 61 65.75 70.5 70.5 75.25 80
#> 81:100 81 85.75 90.5 90.5 95.25 100
ffvecapply(summary(x[i1:i2]), X=x, RETURN=TRUE, CFUN="cmean", BATCHSIZE=20)
#> Min. 1st Qu. Median Mean 3rd Qu. Max.
#> 41.00 45.75 50.50 50.50 55.25 60.00
message("how to do colSums with ffrowapply")
#> how to do colSums with ffrowapply
x <- ff(1:1000, vmode="integer", dim=c(100, 10))
ffrowapply(colSums(x[i1:i2,,drop=FALSE]), X=x, RETURN=TRUE, CFUN="list", BATCHSIZE=20)
#> $`1:20`
#> [1] 210 2210 4210 6210 8210 10210 12210 14210 16210 18210
#>
#> $`21:40`
#> [1] 610 2610 4610 6610 8610 10610 12610 14610 16610 18610
#>
#> $`41:60`
#> [1] 1010 3010 5010 7010 9010 11010 13010 15010 17010 19010
#>
#> $`61:80`
#> [1] 1410 3410 5410 7410 9410 11410 13410 15410 17410 19410
#>
#> $`81:100`
#> [1] 1810 3810 5810 7810 9810 11810 13810 15810 17810 19810
#>
ffrowapply(colSums(x[i1:i2,,drop=FALSE]), X=x, RETURN=TRUE, CFUN="crbind", BATCHSIZE=20)
#> [,1] [,2] [,3] [,4] [,5] [,6] [,7] [,8] [,9] [,10]
#> 1:20 210 2210 4210 6210 8210 10210 12210 14210 16210 18210
#> 21:40 610 2610 4610 6610 8610 10610 12610 14610 16610 18610
#> 41:60 1010 3010 5010 7010 9010 11010 13010 15010 17010 19010
#> 61:80 1410 3410 5410 7410 9410 11410 13410 15410 17410 19410
#> 81:100 1810 3810 5810 7810 9810 11810 13810 15810 17810 19810
ffrowapply(colSums(x[i1:i2,,drop=FALSE]), X=x, RETURN=TRUE, CFUN="csum", BATCHSIZE=20)
#> [1] 5050 15050 25050 35050 45050 55050 65050 75050 85050 95050
message("further ffrowapply examples")
#> further ffrowapply examples
x <- ff(1:1000, vmode="integer", dim=c(100, 10))
message("loop evaluate expression without returning anything")
#> loop evaluate expression without returning anything
ffrowapply(x[i1:i2, ] <- i1:i2, X=x, BATCHSIZE=20)
message("lets return the combined expressions as a new ff object (x unchanged)")
#> lets return the combined expressions as a new ff object (x unchanged)
ffrowapply(2*x[i1:i2, ], X=x, RETURN=TRUE, BATCHSIZE=20)
#> ff (open) integer length=1000 (1000) dim=c(100,10) dimorder=c(1,2)
#> [,1] [,2] [,3] [,4] [,5] [,6] [,7] [,8] [,9] [,10]
#> [1,] 2 2 2 2 2 2 2 2 2 2
#> [2,] 4 4 4 4 4 4 4 4 4 4
#> [3,] 6 6 6 6 6 6 6 6 6 6
#> [4,] 8 8 8 8 8 8 8 8 8 8
#> [5,] 10 10 10 10 10 10 10 10 10 10
#> [6,] 12 12 12 12 12 12 12 12 12 12
#> [7,] 14 14 14 14 14 14 14 14 14 14
#> [8,] 16 16 16 16 16 16 16 16 16 16
#> : : : : : : : : : : :
#> [93,] 186 186 186 186 186 186 186 186 186 186
#> [94,] 188 188 188 188 188 188 188 188 188 188
#> [95,] 190 190 190 190 190 190 190 190 190 190
#> [96,] 192 192 192 192 192 192 192 192 192 192
#> [97,] 194 194 194 194 194 194 194 194 194 194
#> [98,] 196 196 196 196 196 196 196 196 196 196
#> [99,] 198 198 198 198 198 198 198 198 198 198
#> [100,] 200 200 200 200 200 200 200 200 200 200
message("lets return a single row aggregate")
#> lets return a single row aggregate
ffrowapply(t(apply(x[i1:i2,,drop=FALSE], 1, mean)), X=x, RETURN=TRUE, RETCOL=NULL, BATCHSIZE=20)
#> ff (open) integer length=100 (100)
#> [1] [2] [3] [4] [5] [6] [7] [8] [93] [94] [95] [96]
#> 1 2 3 4 5 6 7 8 : 93 94 95 96
#> [97] [98] [99] [100]
#> 97 98 99 100
message("lets return a 6 column aggregates")
#> lets return a 6 column aggregates
y <- ffrowapply( t(apply(x[i1:i2,,drop=FALSE], 1, summary)), X=x
, RETURN=TRUE, RETCOL=length(summary(0)), BATCHSIZE=20)
colnames(y) <- names(summary(0))
y
#> ff (open) integer length=600 (600) dim=c(100,6) dimorder=c(1,2)
#> Min. 1st Qu. Median Mean 3rd Qu. Max.
#> [1,] 1 1 1 1 1 1
#> [2,] 2 2 2 2 2 2
#> [3,] 3 3 3 3 3 3
#> [4,] 4 4 4 4 4 4
#> [5,] 5 5 5 5 5 5
#> [6,] 6 6 6 6 6 6
#> [7,] 7 7 7 7 7 7
#> [8,] 8 8 8 8 8 8
#> : : : : : : :
#> [93,] 93 93 93 93 93 93
#> [94,] 94 94 94 94 94 94
#> [95,] 95 95 95 95 95 95
#> [96,] 96 96 96 96 96 96
#> [97,] 97 97 97 97 97 97
#> [98,] 98 98 98 98 98 98
#> [99,] 99 99 99 99 99 99
#> [100,] 100 100 100 100 100 100
message("determine column minima if a complete column does not fit into RAM")
#> determine column minima if a complete column does not fit into RAM
ffrowapply(apply(x[i1:i2,], 2, min), X=x, RETURN=TRUE, CFUN="pmin", BATCHSIZE=20)
#> [1] 1 1 1 1 1 1 1 1 1 1
message("ffapply examples")
#> ffapply examples
x <- ff(1:720, dim=c(8,9,10))
dimnames(x) <- dummy.dimnames(x)
message("apply function with scalar return value")
#> apply function with scalar return value
apply(X=x[], MARGIN=3:2, FUN=sum)
#> b1 b2 b3 b4 b5 b6 b7 b8 b9
#> c1 36 100 164 228 292 356 420 484 548
#> c2 612 676 740 804 868 932 996 1060 1124
#> c3 1188 1252 1316 1380 1444 1508 1572 1636 1700
#> c4 1764 1828 1892 1956 2020 2084 2148 2212 2276
#> c5 2340 2404 2468 2532 2596 2660 2724 2788 2852
#> c6 2916 2980 3044 3108 3172 3236 3300 3364 3428
#> c7 3492 3556 3620 3684 3748 3812 3876 3940 4004
#> c8 4068 4132 4196 4260 4324 4388 4452 4516 4580
#> c9 4644 4708 4772 4836 4900 4964 5028 5092 5156
#> c10 5220 5284 5348 5412 5476 5540 5604 5668 5732
apply(X=x[], MARGIN=2:3, FUN=sum)
#> c1 c2 c3 c4 c5 c6 c7 c8 c9 c10
#> b1 36 612 1188 1764 2340 2916 3492 4068 4644 5220
#> b2 100 676 1252 1828 2404 2980 3556 4132 4708 5284
#> b3 164 740 1316 1892 2468 3044 3620 4196 4772 5348
#> b4 228 804 1380 1956 2532 3108 3684 4260 4836 5412
#> b5 292 868 1444 2020 2596 3172 3748 4324 4900 5476
#> b6 356 932 1508 2084 2660 3236 3812 4388 4964 5540
#> b7 420 996 1572 2148 2724 3300 3876 4452 5028 5604
#> b8 484 1060 1636 2212 2788 3364 3940 4516 5092 5668
#> b9 548 1124 1700 2276 2852 3428 4004 4580 5156 5732
ffapply(X=x, MARGIN=3:2, AFUN="sum", RETURN=TRUE, BATCHSIZE=8)
#> ff (open) integer length=90 (90) dim=c(10,9) dimorder=c(1,2)
#> [,1] [,2] [,3] [,4] [,5] [,6] [,7] [,8] [,9]
#> [1,] 36 100 164 228 292 356 420 484 548
#> [2,] 612 676 740 804 868 932 996 1060 1124
#> [3,] 1188 1252 1316 1380 1444 1508 1572 1636 1700
#> [4,] 1764 1828 1892 1956 2020 2084 2148 2212 2276
#> [5,] 2340 2404 2468 2532 2596 2660 2724 2788 2852
#> [6,] 2916 2980 3044 3108 3172 3236 3300 3364 3428
#> [7,] 3492 3556 3620 3684 3748 3812 3876 3940 4004
#> [8,] 4068 4132 4196 4260 4324 4388 4452 4516 4580
#> [9,] 4644 4708 4772 4836 4900 4964 5028 5092 5156
#> [10,] 5220 5284 5348 5412 5476 5540 5604 5668 5732
message("this is what CFUN is based on")
#> this is what CFUN is based on
ffapply(X=x, MARGIN=2:3, AFUN="sum", RETURN=TRUE, CFUN="list", BATCHSIZE=8)
#> 1:1 2:2 3:3 4:4 5:5 6:6 7:7 8:8 9:9 10:10
#> 1:1 36 612 1188 1764 2340 2916 3492 4068 4644 5220
#> 2:2 100 676 1252 1828 2404 2980 3556 4132 4708 5284
#> 3:3 164 740 1316 1892 2468 3044 3620 4196 4772 5348
#> 4:4 228 804 1380 1956 2532 3108 3684 4260 4836 5412
#> 5:5 292 868 1444 2020 2596 3172 3748 4324 4900 5476
#> 6:6 356 932 1508 2084 2660 3236 3812 4388 4964 5540
#> 7:7 420 996 1572 2148 2724 3300 3876 4452 5028 5604
#> 8:8 484 1060 1636 2212 2788 3364 3940 4516 5092 5668
#> 9:9 548 1124 1700 2276 2852 3428 4004 4580 5156 5732
message("apply functions with vector or array return value currently have limited support")
#> apply functions with vector or array return value currently have limited support
apply(X=x[], MARGIN=3:2, FUN=summary)
#> , , b1
#>
#> c1 c2 c3 c4 c5 c6 c7 c8 c9 c10
#> Min. 1.00 73.00 145.00 217.00 289.00 361.00 433.00 505.00 577.00 649.00
#> 1st Qu. 2.75 74.75 146.75 218.75 290.75 362.75 434.75 506.75 578.75 650.75
#> Median 4.50 76.50 148.50 220.50 292.50 364.50 436.50 508.50 580.50 652.50
#> Mean 4.50 76.50 148.50 220.50 292.50 364.50 436.50 508.50 580.50 652.50
#> 3rd Qu. 6.25 78.25 150.25 222.25 294.25 366.25 438.25 510.25 582.25 654.25
#> Max. 8.00 80.00 152.00 224.00 296.00 368.00 440.00 512.00 584.00 656.00
#>
#> , , b2
#>
#> c1 c2 c3 c4 c5 c6 c7 c8 c9 c10
#> Min. 9.00 81.00 153.00 225.00 297.00 369.00 441.00 513.00 585.00 657.00
#> 1st Qu. 10.75 82.75 154.75 226.75 298.75 370.75 442.75 514.75 586.75 658.75
#> Median 12.50 84.50 156.50 228.50 300.50 372.50 444.50 516.50 588.50 660.50
#> Mean 12.50 84.50 156.50 228.50 300.50 372.50 444.50 516.50 588.50 660.50
#> 3rd Qu. 14.25 86.25 158.25 230.25 302.25 374.25 446.25 518.25 590.25 662.25
#> Max. 16.00 88.00 160.00 232.00 304.00 376.00 448.00 520.00 592.00 664.00
#>
#> , , b3
#>
#> c1 c2 c3 c4 c5 c6 c7 c8 c9 c10
#> Min. 17.00 89.00 161.00 233.00 305.00 377.00 449.00 521.00 593.00 665.00
#> 1st Qu. 18.75 90.75 162.75 234.75 306.75 378.75 450.75 522.75 594.75 666.75
#> Median 20.50 92.50 164.50 236.50 308.50 380.50 452.50 524.50 596.50 668.50
#> Mean 20.50 92.50 164.50 236.50 308.50 380.50 452.50 524.50 596.50 668.50
#> 3rd Qu. 22.25 94.25 166.25 238.25 310.25 382.25 454.25 526.25 598.25 670.25
#> Max. 24.00 96.00 168.00 240.00 312.00 384.00 456.00 528.00 600.00 672.00
#>
#> , , b4
#>
#> c1 c2 c3 c4 c5 c6 c7 c8 c9 c10
#> Min. 25.00 97.00 169.00 241.00 313.00 385.00 457.00 529.00 601.00 673.00
#> 1st Qu. 26.75 98.75 170.75 242.75 314.75 386.75 458.75 530.75 602.75 674.75
#> Median 28.50 100.50 172.50 244.50 316.50 388.50 460.50 532.50 604.50 676.50
#> Mean 28.50 100.50 172.50 244.50 316.50 388.50 460.50 532.50 604.50 676.50
#> 3rd Qu. 30.25 102.25 174.25 246.25 318.25 390.25 462.25 534.25 606.25 678.25
#> Max. 32.00 104.00 176.00 248.00 320.00 392.00 464.00 536.00 608.00 680.00
#>
#> , , b5
#>
#> c1 c2 c3 c4 c5 c6 c7 c8 c9 c10
#> Min. 33.00 105.00 177.00 249.00 321.00 393.00 465.00 537.00 609.00 681.00
#> 1st Qu. 34.75 106.75 178.75 250.75 322.75 394.75 466.75 538.75 610.75 682.75
#> Median 36.50 108.50 180.50 252.50 324.50 396.50 468.50 540.50 612.50 684.50
#> Mean 36.50 108.50 180.50 252.50 324.50 396.50 468.50 540.50 612.50 684.50
#> 3rd Qu. 38.25 110.25 182.25 254.25 326.25 398.25 470.25 542.25 614.25 686.25
#> Max. 40.00 112.00 184.00 256.00 328.00 400.00 472.00 544.00 616.00 688.00
#>
#> , , b6
#>
#> c1 c2 c3 c4 c5 c6 c7 c8 c9 c10
#> Min. 41.00 113.00 185.00 257.00 329.00 401.00 473.00 545.00 617.00 689.00
#> 1st Qu. 42.75 114.75 186.75 258.75 330.75 402.75 474.75 546.75 618.75 690.75
#> Median 44.50 116.50 188.50 260.50 332.50 404.50 476.50 548.50 620.50 692.50
#> Mean 44.50 116.50 188.50 260.50 332.50 404.50 476.50 548.50 620.50 692.50
#> 3rd Qu. 46.25 118.25 190.25 262.25 334.25 406.25 478.25 550.25 622.25 694.25
#> Max. 48.00 120.00 192.00 264.00 336.00 408.00 480.00 552.00 624.00 696.00
#>
#> , , b7
#>
#> c1 c2 c3 c4 c5 c6 c7 c8 c9 c10
#> Min. 49.00 121.00 193.00 265.00 337.00 409.00 481.00 553.00 625.00 697.00
#> 1st Qu. 50.75 122.75 194.75 266.75 338.75 410.75 482.75 554.75 626.75 698.75
#> Median 52.50 124.50 196.50 268.50 340.50 412.50 484.50 556.50 628.50 700.50
#> Mean 52.50 124.50 196.50 268.50 340.50 412.50 484.50 556.50 628.50 700.50
#> 3rd Qu. 54.25 126.25 198.25 270.25 342.25 414.25 486.25 558.25 630.25 702.25
#> Max. 56.00 128.00 200.00 272.00 344.00 416.00 488.00 560.00 632.00 704.00
#>
#> , , b8
#>
#> c1 c2 c3 c4 c5 c6 c7 c8 c9 c10
#> Min. 57.00 129.00 201.00 273.00 345.00 417.00 489.00 561.00 633.00 705.00
#> 1st Qu. 58.75 130.75 202.75 274.75 346.75 418.75 490.75 562.75 634.75 706.75
#> Median 60.50 132.50 204.50 276.50 348.50 420.50 492.50 564.50 636.50 708.50
#> Mean 60.50 132.50 204.50 276.50 348.50 420.50 492.50 564.50 636.50 708.50
#> 3rd Qu. 62.25 134.25 206.25 278.25 350.25 422.25 494.25 566.25 638.25 710.25
#> Max. 64.00 136.00 208.00 280.00 352.00 424.00 496.00 568.00 640.00 712.00
#>
#> , , b9
#>
#> c1 c2 c3 c4 c5 c6 c7 c8 c9 c10
#> Min. 65.00 137.00 209.00 281.00 353.00 425.00 497.00 569.00 641.00 713.00
#> 1st Qu. 66.75 138.75 210.75 282.75 354.75 426.75 498.75 570.75 642.75 714.75
#> Median 68.50 140.50 212.50 284.50 356.50 428.50 500.50 572.50 644.50 716.50
#> Mean 68.50 140.50 212.50 284.50 356.50 428.50 500.50 572.50 644.50 716.50
#> 3rd Qu. 70.25 142.25 214.25 286.25 358.25 430.25 502.25 574.25 646.25 718.25
#> Max. 72.00 144.00 216.00 288.00 360.00 432.00 504.00 576.00 648.00 720.00
#>
message("you must use CFUN, the rest is up to you")
#> you must use CFUN, the rest is up to you
y <- ffapply(X=x, MARGIN=3:2, AFUN="summary", RETURN=TRUE, CFUN="list", BATCHSIZE=8)
y
#> 1:1 2:2 3:3 4:4 5:5 6:6 7:7
#> 1:1 numeric,6 numeric,6 numeric,6 numeric,6 numeric,6 numeric,6 numeric,6
#> 2:2 numeric,6 numeric,6 numeric,6 numeric,6 numeric,6 numeric,6 numeric,6
#> 3:3 numeric,6 numeric,6 numeric,6 numeric,6 numeric,6 numeric,6 numeric,6
#> 4:4 numeric,6 numeric,6 numeric,6 numeric,6 numeric,6 numeric,6 numeric,6
#> 5:5 numeric,6 numeric,6 numeric,6 numeric,6 numeric,6 numeric,6 numeric,6
#> 6:6 numeric,6 numeric,6 numeric,6 numeric,6 numeric,6 numeric,6 numeric,6
#> 7:7 numeric,6 numeric,6 numeric,6 numeric,6 numeric,6 numeric,6 numeric,6
#> 8:8 numeric,6 numeric,6 numeric,6 numeric,6 numeric,6 numeric,6 numeric,6
#> 9:9 numeric,6 numeric,6 numeric,6 numeric,6 numeric,6 numeric,6 numeric,6
#> 10:10 numeric,6 numeric,6 numeric,6 numeric,6 numeric,6 numeric,6 numeric,6
#> 8:8 9:9
#> 1:1 numeric,6 numeric,6
#> 2:2 numeric,6 numeric,6
#> 3:3 numeric,6 numeric,6
#> 4:4 numeric,6 numeric,6
#> 5:5 numeric,6 numeric,6
#> 6:6 numeric,6 numeric,6
#> 7:7 numeric,6 numeric,6
#> 8:8 numeric,6 numeric,6
#> 9:9 numeric,6 numeric,6
#> 10:10 numeric,6 numeric,6
y[[1]]
#> , , b1
#>
#> c1
#> Min. 1.00
#> 1st Qu. 2.75
#> Median 4.50
#> Mean 4.50
#> 3rd Qu. 6.25
#> Max. 8.00
#>
rm(x); gc()
#> used (Mb) gc trigger (Mb) max used (Mb)
#> Ncells 1145348 61.2 1994352 106.6 1994352 106.6
#> Vcells 2127936 16.3 8388608 64.0 3981144 30.4