I want a high-end intercheangeable lens light field camera and I want it now. But the resolution, you tell me. But the bandwidth. First, it’s clear that if you look at the pixel size for phone cameras and you make a full-frame sensor out of it, you can get all the resolution you need (a 24x36mm sensor at 1 micron gives you 864MPixels). But let’s say I am reasonable and only ask for, say a 120MPixels full frame sensor. It’s only twice the number of pixels as the Sony A7r4 and Canon has made a prototype of a 120Mpixel sensor ten years ago. Even with the current bandwidth (the Sony can shoot at 10fps), I would be able to shoot at 5fps, not so bad unless you’re doing some serious sport photography.

Let’s say I leverage my 120MPixels to put 3×3 microlenses. My spatial resolution becomes 13 MPixels, which is really not bad (and you can probably reconstruct higher-resolution images with light field superresolution tricks, let’s say 20MPixels).  With these 3×3 aperture sub-pixels, I can 1/ correct for optical aberration (chromatic and spherical in particular, but also field curvature) and 2/ refocus by a factor of roughly 3x, meaning if I shoot at f/2.8 and refocus within the depth of field of an f/8 lens. This is huge.

Now one challenge with 3×3 is that which photosites get light depends on the aperture of the lens. It’s fine if you only have f/2.8 lenses and always shoot at f/2.8, otherwise the periphery aperture sub-pixels get less light and you’re wasting them. One option around this is to only do 2×2, which is only a factor of 2 compared to Canon dual pixel autofocus. This may end up needed for AF purposes anyway because, currently, Canon’s on-sensor AF is only sensitive to horizontal contrast (unlike the cross-type AF points of good DSLRs). Then with these quad pixels, we could refocus by a factor of 2, which is already not so bad (refocus with f/5.6 when shooting at f/2.8). It’s not the kind of dramatic refocusing you see in academic light field papers, but it can fix a lot of situations when, e.g., the focus is on the nose rather than the eyes.