Barrel distortion in WebGL

So after some chats it's become quite apparent that the resources spent on WebVR are super small. Both from Google and from Mozilla's side. It's also clear that there'll be no resources to put effort into getting the magic happen for 2d canvas. So I went out to create a simple poc for WebGL that applies the barrel distortion to a 2d canvas.

While I was hoping for some quick copy/paste magic, I quickly found myself learning the basics of WebGL. In part because nearly all the examples direct you to use ThreeJS and in part because none of the other examples were copy/pastable enough to work out of the box.

As an aside, let me rant a little on that notion. First of, I have the greatest respect for Mr.Doob. No seriously, this will have nothing to do with him or the people on the project. That said, it feels like ThreeJS has become the same as jQuery / Angular. You can't do a search for bare bone JavaScript examples on something without encountering at least five different jQuery examples (jQuery seems to be more prevelant than Angular in this). Even for trivial DOM stuff you'll get dozens of library answers before getting the bare bone answer. Now I don't mind libraries at all. I do mind it if they take on their own life as a pseudo language and people don't even know anymore what's actually going on. Or well, that may work for some, but I really want to know what's going on in the language itself. So when venturing out to get this WebGL code working I have the same problem. For anything you search on you get dozens of ThreeJS examples. I have to explicitly exclude the term from my query in order to get something I can work with. So frustrating. Makes me think many people that "know webgl" actually don't have the slightest idea... Very similar to people that "program in jQuery". *shudder* To each their own.

Okay so this is kind of like a tutorial that leads from a simple example to a function that takes a 2d canvas, renders it as a texture in WebGL, and applies the barrel distortion to it through a shader. The end result is not optimal, by far. I'm certain I'm doing all kinds of things wrong here, by anyone's standard. But it works sufficiently for my purpose right now. So for the result, scroll doooown.

None of these code examples use any library. Single page with everything inline. I'll try to explain what's doing what, why, where, as far as I understand it myself. If you've done anything in WebGL you'll be very bored, and probably slightly annoyed by the mistakes ;) Ohi myself in five years! I'm sorry!

In the beginning

I've saved various stages of my process. We'll start off with the simplest example. We'll paint a rectangle on screen:

<canvas id="canvas" width="400" height="300"></canvas>
<script type="vertex">
attribute vec2 a_pos;
varying vec2 v_pos;
void main() {
v_pos = vec2(a_pos.xy);
gl_Position = vec4(a_pos, 0, 1);
}
</script>
<script type="fragment">
precision highp float;
varying vec2 v_pos;
void main() {
gl_FragColor = vec4(v_pos, 0.0, 1.0);
}
</script>
<script>
var canvas = document.getElementById("canvas");
var gl = canvas.getContext('experimental-webgl');
if (!gl) throw 'webgl fail';

try {
var program = gl.createProgram();

// vertex shader: pixel mapping set vec4 to gl_Position
var vertShaderObj = gl.createShader(gl.VERTEX_SHADER);
var vertexShaderSrc = document.querySelector('[type="vertex"]').textContent;
gl.shaderSource(vertShaderObj, vertexShaderSrc);
gl.compileShader(vertShaderObj);
gl.attachShader(program, vertShaderObj);

// fragment shader: pixel coloring, set vec4 to gl_FragColor
var fragShaderObj = gl.createShader(gl.FRAGMENT_SHADER);
var fragmentShaderSrc = document.querySelector('[type="fragment"]').textContent;
gl.shaderSource(fragShaderObj, fragmentShaderSrc);
gl.compileShader(fragShaderObj);
gl.attachShader(program, fragShaderObj);

gl.linkProgram(program);
gl.useProgram(program);
if (!gl.getShaderParameter(vertShaderObj, gl.COMPILE_STATUS)) throw new Error("Could not compile shader: " + gl.getShaderInfoLog(vertShaderObj));
if (!gl.getShaderParameter(fragShaderObj, gl.COMPILE_STATUS)) throw new Error("Could not compile shader: " + gl.getShaderInfoLog(fragShaderObj));
} catch(e) {
console.error(e && e.stack || e);
}

var positionLocation = gl.getAttribLocation(program, "a_pos");

var buffer = gl.createBuffer();
gl.bindBuffer(gl.ARRAY_BUFFER, buffer);
gl.bufferData(gl.ARRAY_BUFFER, new Float32Array([
// two rectangles
-1.0, -1.0,
1.0, -1.0,
-1.0, 1.0,

-1.0, 1.0,
1.0, -1.0,
1.0, 1.0]), gl.STATIC_DRAW);
gl.enableVertexAttribArray(positionLocation);
gl.vertexAttribPointer(positionLocation, 2, gl.FLOAT, false, 0, 0);

// draw
gl.drawArrays(gl.TRIANGLES, 0, 6);
</script>

This should render you a nice rectangle with gradient.

In short the code paints two triangles. It uses the x and y position of each pixel as the r and g value when determining the color. The colors inside the triangle are extrapolated (we only instruct the corner tips of each triangle). The result is a rectangle with gradient.

So from the top:

There's a canvas object which is used to render everything.

There are two "weird" script tags. They have an arbitrary type value. The point here is to have any type that's not recognized as "JavaScript" by some browser. This allows you to put invisible multiline string content in the document. We can read out that content later. It looks better than having string concats, or even "multiline JavaScript strings". So these script tags are nothing special at all, just a generic trick to get arbitrary text content into JS.

We see two of these scripts. Their content belongs to two different "shaders" which is what we use in WebGL to do the magic. Now I'm sure I'll clobber some definitions here, but the way I see it a shader allows you to map input pixels to output pixels and input pixels to output colors. We have two shaders here; the "vertex" shader, which maps input to pixel locations, and the "fragment" shader, which maps pixels to colors.

There are four basic "varible" types in shaders. I'm not entirely down on them yet, but from what I understand you have a "uniform", which you can set from JS. They are read-only in the shader. There's an "attribute", which is still a bit weird for me, but it's supposed to get certain attributes of a vertex (point). There's the "varying" type, which is the only type that allows you to send data from the vertex shader to the fragment shader. It feels a bit verbose to me, but I'm sure it has its reasons. Other than that, the varying type can be of any type, hence it's name. I guess the compiler determines the actual type at compile time, which makes sense if you can't set it from JS anyways. There's also "const", which allows you to define constants.

There's a bunch of types. I won't cover them all, only those we use. There's "float" for a floating point. You use "vec2", "vec3", and "vec4" for vectors, which is basically an ordered set of floats like [1.0, 0.5, 1.0, 2.4] for vec4. It's not actually an array at all though! Just think of vectors as ordered numbers. Their meaning depends on the context. Could be xyz or rgba or whatever. They're just numbers. Obviously there are more types, but let's save that for later.

In the fragment shader you see precision highp float;, which is something I had to add to get the shader to compile. Honestly I don't know why, none of the examples I've seen used it, but whatever. I'm happy <ondras> on irc helped me figuring out that one.

The JS starts with trivial canvas bootstrapping stuff. Getting the canvas, getting the gl context. Where you tend to name the context variable ctx or even just c for 2d canvas, in WebGL it's customary to use gl or g.

First we'll compile the shaders into a "program". We'll get the two shaders and compile them into the program:

var vertShaderObj = gl.createShader(gl.VERTEX_SHADER);
var vertexShaderSrc = document.querySelector('[type="vertex"]').textContent;
gl.shaderSource(vertShaderObj, vertexShaderSrc);
gl.compileShader(vertShaderObj);
gl.attachShader(program, vertShaderObj);

See how I merely retrieve the fake script content for text? Could have inlined it as a string literal all the same. There's little to explain about these lines of code. You need them and they perform the compilation magic. Twice.

At the end of the compilation there's a check to see whether the shaders were compiled at all. You can use something like if (!gl.getShaderParameter(vertShaderObj, gl.COMPILE_STATUS)) console.error("Could not compile shader: " + gl.getShaderInfoLog(vertShaderObj)); to get more information on why a shader did not compile. For me, Firefox was much more helpful by default than Chrome was. And even with the info logs, Chrome would sometimes not report anything while Firefox would. So as far as debugging goes, Firefox wins it right now.

The main functions of the shaders have no return value. I guess there's no return keyword in GLSL, the shader language. The main function must at some point set a variable which is picked up by WebGL. I would call that the return value for now, though of course there are many more such "live" variables you could set. Each shader has one that's most pertinent though. The vertex shader should set gl_Position, to tell WebGL which pixel of the canvas you're paiting right now (note that this can be anywhere in the -1.0 ... 1.0 range, regardless of the input). The fragment shader must set gl_FragColor, which sets the rgba color of that pixel. Again, this can be anything inside the 0.0 ... 1.0 range.

It took me a while to figure out that the main functions don't actually return a value but that updating these variables is what makes the program work. I dunno, not very intuitive from where I'm coming from. But once you know ... :)

Take a look at var positionLocation = gl.getAttribLocation(program, "a_pos");. I interpret this as getting the byte offset for the a_pos variable in the compiled program. We'll need that to set its value. As I said, I'm not entirely sure yet when you would use an attribute and when you'd use a uniform, but in this case it's an attribute.

The next bunch creates a buffer of vertexes (points) to use as paint parameters later. You put them in a buffer and basically tell WebGL how to interpret them.

gl.bufferData(gl.ARRAY_BUFFER, new Float32Array([
// two rectangles
-1.0, -1.0,
1.0, -1.0,
-1.0, 1.0,

-1.0, 1.0,
1.0, -1.0,
1.0, 1.0]), gl.STATIC_DRAW);

This "line" of code puts the coordinates of two rectangles into the buffer. Screen coordinates run from -1 to 1, top-left to bottom-right (IIRC). So the 12 values above are actually six vertex coordinates (<-1, -1>, <1, -1>, <-1, 1>, <-1, 1>, <1, -1>, <1, 1>) which make up for two times three corners of a triangle, covering the entire viewport. The STATIC_DRAW is an optimizer argument, not relevant for us here.

Next we'll tell WebGL how to use these points:

gl.enableVertexAttribArray(positionLocation);
gl.vertexAttribPointer(positionLocation, 2, gl.FLOAT, false, 0, 0);

It'll bind each point to the a_pos, which was a vec2 so it fits two floats. The second line tells WebGL to step through the buffer two elements at a time and interpret them as floats.

The last line, gl.drawArrays(gl.TRIANGLES, 0, 6);, is where the painting happens. It tells WebGL to paint triangles. This model will consume three vectors per triangle. It should start at index 0 in the buffer and run up to but not including index 6. There are a few possible types you can do here besides TRIANGLES. For example, TRIANGLE_FAN draws triangles but each vector after the first three uses the last two vectors to create another triangle (so you get a fan of them). POINTS will draw the vectors as single pixels. Etc.

So what happens? Our program actually kinda starts at the last line. The drawArrays method will start running through the buffer we've prepared. It'll take three x,y coordinates from the buffer and paint a triangle with them. When it does it will execute the main function of each shader for each of the coordinates. The vertex shader only passes on this coordinate to gl_Position directly (setting the z to 0.0 and w to 1.0). It also puts the coordinate into a varying variable so the fragment shader can use it.

The fragment shader uses the input coordinate to set gl_FragColor to an rgba value by using the vec2 coordinates and creating a vec4 from them, which acts as an rgba value (r=x, g=y, b=0, a=1).

We only actually instruct WebGL to paint those six points (three for each triangle). We'll tell it which color it gets and it will extrapolate the colors between which creates the gradient. Since we'll paint two such triangles we can fill the entire viewport. You actually never really paint rectangles, everything is composed of triangles. I forgot exactly why this was, but I remember it comes down to making everything simpler behind the scenes. Super optimized triangle matrix compuations.

Alright, here's the result if your browser supports WebGL:

Texture sampling

Hopefully you got no errors there :)

The next step was to be able to use pixels from the canvas. We could paint the canvas directly on screen using two triangles and the canvas as a texture, but that won't apply the distortion we need. So we'll construct the canvas as a texture and "sample" of that instead. This samplign is basically "tasting" the colors at certain indexes using certain techniques. I'll concede immediately that there's probably a better way to do the sampling, as mine is very lossy, but I did not need to do that for now so the input image will be clobbered.

First we'll create the input canvas. We'll do a trivial rotating square for the animation effect. This should not be anything magical.

<canvas id="B"></canvas>
<script>
var w, h = w = Math.pow(2, 9);
var leftCanvas = document.querySelector('#B');
leftCanvas.width = leftCanvas.height = w;
var ctx = leftCanvas.getContext('2d');
var r = 0;
(function yy() {
requestAnimationFrame(yy);

ctx.fillStyle = 'blue';
ctx.fillRect(0, 0, leftCanvas.width, leftCanvas.height);
ctx.save();
ctx.beginPath();
ctx.translate(250, 250);
ctx.rotate((++r * 0.05) % 2*Math.PI);
ctx.fillStyle = 'red';
ctx.fillRect(0, 0, 75, 75);
ctx.restore();
})();
</script>

Get the canvas, paint it blue, save translate and rotate it, paint a red rect, restore it. Repeat for animation.

We now need to put this canvas as a texture in our WebGL program. In the fragment shader we'll create a new variable: uniform sampler2D u_tex;. The sampler2D type allows us to put a texture there. In the vertex shader we'll add gl_PointSize = 1.0; which affects the sample size, and therefor quality. The fragment shader now "outputs" it's result by calling a sampler function: gl_FragColor = texture2D(uSampler, v_pos);. The rest remains the same so our shaders now look like this:

<script type="vertex">
attribute vec2 a_pos;
varying vec2 v_pos;
void main() {
v_pos = vec2(a_pos.xy);
gl_Position = vec4(a_pos.xy, 0.0, 1.0);
gl_PointSize = 1.0;
}
</script>
<script type="fragment">
precision highp float;

varying vec2 v_pos;
uniform sampler2D uSampler;

void main() {
gl_FragColor = texture2D(uSampler, v_pos);
}
</script>

The JS code starts out the same. Bootstrap and compilation are the same as before. The rest of the code looks like this:

var n = 0;
var arr = [];
for (var i=0; i<512*512*2; i+=2) {
arr[ i] = (Math.floor(++n%512) / 256) - 1;
arr[i+1] = (Math.round(n/512) / 256) - 1;
}
var farr = new Float32Array(arr);

var positionLocation = gl.getAttribLocation(program, 'a_pos');
var buffer = gl.createBuffer();
gl.bindBuffer(gl.ARRAY_BUFFER, buffer);
gl.bufferData(gl.ARRAY_BUFFER, farr, gl.STATIC_DRAW);
gl.enableVertexAttribArray(positionLocation);
gl.vertexAttribPointer(positionLocation, 2, gl.FLOAT, false, 0, 0);

(function xx() {
requestAnimationFrame(xx);

var tex = gl.createTexture();
gl.bindTexture(gl.TEXTURE_2D, tex);
gl.texImage2D(gl.TEXTURE_2D, 0, gl.RGBA, gl.RGBA, gl.UNSIGNED_BYTE, leftCanvas);
gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_MIN_FILTER, gl.LINEAR);

gl.activeTexture(gl.TEXTURE0);
gl.bindTexture(gl.TEXTURE_2D, tex);
gl.uniform1i(gl.getUniformLocation(program, 'u_tex'), 0);

// draw
gl.drawArrays(gl.POINTS, 0, arr.length/2);

tex.deallocate();

We start by creating the buffer with coordinates. I'm pretty sure this could be done much more efficiently, but at least it's something we can cache. We only create the buffer once. As you can see it's creating 512*512*2 floats. (You should cache that multiplication ;)) For each pixel, create the corresponding x,y coordinates on the texture (the canvas) normalized to -1.0 ... 1.0. Stash this array in a Float32Array for speed and typing.

Next we'll bind the a_pos to these coordinates like we have before. We tell it each such coordinate consists of two floats. If we were using 3d space we could add another float to the buffer and tell it to use three floats instead. But since we're using a 2d canvas, there's no need.

So far the stuff we can cache. Since we'll use a changing canvas, we'll have to create a new texture for it every step. We requestAnimationFrame for creating the animation.

First create the texture:

var tex = gl.createTexture();
gl.bindTexture(gl.TEXTURE_2D, tex);
gl.texImage2D(gl.TEXTURE_2D, 0, gl.RGBA, gl.RGBA, gl.UNSIGNED_BYTE, leftCanvas);
gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_MIN_FILTER, gl.LINEAR);

If we'd use this image as an actual texture, like a wall or floor tile, we could cache this texture of course. But we don't, so bind it to a shader variable:

gl.activeTexture(gl.TEXTURE0);
gl.bindTexture(gl.TEXTURE_2D, tex);
gl.uniform1i(gl.getUniformLocation(program, 'u_tex'), 0);

And make sure to delete it asap with tex.deallocate();, after painting, because otherwise you'll blow out your GPU (*I mean crash, nothing permanent*).

The actual drawArrays call now gets a gl.POINTS parameter. So each coordinate in our buffer is painted as a single pixel.

So this example will create and cache a buffer with one coordinate for each pixel in the source canvas. For each frame painted it will copy the source canvas to a texture. It will put that texture in a variable for the shader to use as a sample source. The vertex shader doesn't do that much, mainly passes on the current coordinate and affects the quality.

The fragment shader determines the color of the current coordinate by sampling the texture, our source canvas, and using the resulting color from that sample as the pixel color. This will cause some fading due to the sampling, but that's not important for me right now.

So this is the result (top: 2d, bottom: webgl):

Coordinate space

But wait... that's not right? Correct!

You're seeing only a single quadrant used. This is caused by normalizing to the wrong range. All we have to do is fix our buffer and the entire viewport will be used:

var n = 0;
var arr = [];
for (var i = 0; i < source.width * source.height * 2; i += 2) {
arr[i ] = (Math.floor(++n % source.width) / source.width) * 2 - 1;
arr[i + 1] = (Math.round(n / source.width) / source.height) * 2 - 1;
}
var farr = new Float32Array(arr);


Texture coordinates

Okay so now we've filled the screen. But we're seeing quads now. The canvas is mirrored four times on screen. What the..?

This is caused by wrap around reading from the texture. By default it will mod the coordinates with the width/height of the texture. So reading the 5th pixel of a 4 pixel texture will read the first. You can disable that behavior, but for our case we need to fix our coordinate. In the previous step we've converted our 0 .. 1 range to -1 .. 1. This means we're also reading that range on the texture. That's causing the quadruple image. So for the texture we'll need to undo that part conversion. We'll just be lazy and do it in the shader...

<script type="vertex">
attribute vec2 a_pos;
varying vec2 v_pos;
void main() {
v_pos = (a_pos - 1.0) * 0.5;
gl_Position = vec4(a_pos, 0.0, 1.0);
gl_PointSize = 1.0;
}
</script>

Which should get rid of the quads:

Inverted

The result looks nice but it's flipped and mirrored. Luckily there's an easy fix for that. We apply gl.pixelStorei(gl.UNPACK_FLIP_Y_WEBGL, true); to fix that:

var tex = gl.createTexture();
gl.bindTexture(gl.TEXTURE_2D, tex);
gl.texImage2D(gl.TEXTURE_2D, 0, gl.RGBA, gl.RGBA, gl.UNSIGNED_BYTE, canvas);
gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_MIN_FILTER, gl.LINEAR);
gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_MAG_FILTER, gl.LINEAR);
gl.pixelStorei(gl.UNPACK_FLIP_Y_WEBGL, true);

Which yields us:

Applying barrel distortion

So now we're ready to apply the barrel distortion. In short we're going to map the original position of a pixel to a new position as per the barrel distortion effect. This translation is static but we'll use a function for it for now.

precision highp float;

uniform float u_barrel_power;
vec2 distort(vec2 p)
{
float theta = atan(p.y, p.x);
float radius = length(p);
radius = pow(radius, u_barrel_power);
p.x = radius * cos(theta);
p.y = radius * sin(theta);
return 0.5 * (p + 1.0);
}

varying vec2 v_pos;
uniform sampler2D u_tex;

void main() {
gl_FragColor = texture2D(u_tex, distort(v_pos));
}

And in the JS we set the barrel strength:

var barrelPowerLocation = gl.getUniformLocation(program, "BarrelPower"); // gl.getAttribLocation(program, "BarrelPower");
gl.uniform1f(barrelPowerLocation, 1.5);

I got this function pretty much straight from prideout.net. I've added some wireframes to make the effect more obvious. Applying this makes the shader look like:

Clipping

We can apply some texture coordinate clipping while sampling. There are a few options for clamping the coords (like gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_WRAP_T, gl.CLAMP_TO_EDGE);), but none seem to be able to set a default color in case of an OOB so we'll just do it manually in the shader instead:

void main() {
vec2 d = distort(v_pos);
if (d.x > 1.0 || d.x < 0.0 || d.y > 1.0 || d.y < 0.0) gl_FragColor = vec4(1.0, 1.0, 1.0, 1.0); // white
else gl_FragColor = texture2D(u_tex, d);
}


Results in this:

Conclusion

I suck at WebGL. But I learned a lot trying to come up with this experiment. Hope it was useful for you :) Or maybe you got a good laugh out of it ;)

I'm open to suggestions on how to improve this, because I'm certain there are many things I'm doing naievely wrong. You know where to find me.