Additional heightmap compression.

At this rate, I may need to start up the old development log again.


Today, I am working on getting the clipmaps to scroll.  The neat trick here is that as you move around, the map stays still, so you only need to change the part you are using.  Like this.


It actually works quite nicely for the gpu, because one of the options you can set for reading a texture is wrapping on the x and y axis.  Normally, you do this when you have a repeating texture, only here your grid is sized so that each textel is represented exactly once.  Really, the only tricky part, is updating the thing, and while the math isn't terribly difficult, you have to be careful to get the update ranges in just the right spot.

The original NVIDIA presentation on the technique writes to the texture by rendering a pair of quads to the texture.  Back in 2004 when it was written, that was the only way to partially load a texture.  Instead of that, I am using glTexSubImage2D(), which lets you write directly to the texture memory, and additionally lets you specify a subregion to write to.  Another use of the sub region might be adding sprites to an atlas, or setting up a lookup table.  I have has some problems with this function, though.  When I tried loading the full 8192x8192 image for the world map, it worked on my GTX 560 Ti, but crashed on the Radeon HD 6450 I have at work.  The fix was to split it over 4 writes.  Apparently there is a limit to how much data can be uploaded in one go, though I haven't seen it discussed elsewhere.

-------------------
What I've been doing now is trying to organize my texture management.  Up until now it's been handled pretty informally, with all the code in a single file and the settings hardcoded in.  Instead, it will be handled by a terrain manager which not only stores the texture data and updates, but also keeps track of the coordinates and dimensions for each detail level.  In the future it will be much easier to tweak each level individually.

I'm postponing DXT1 compression support for the time being.  The format stores data in 4x4 texel blocks, which must be moved as a unit.  This complicates shifting the grid, so I want to nail down the mesh resolution before tackling that, and this includes hardware tessellation, too.  For now, I'll just use raw colors.

--------------------
I'm thinking that for the height field, I want at least the first level to match the resolution of the color map, which would be another 1MB.  I'd also like to look into having a secondary height field specifically to match the resolution of the mesh grid.  Since the mesh needs to completely envelop the virtual geometry, you can't simply use the average height value.  If your mesh vertices are scaled 1:1, you can, however, take the max height of the 4 columns the vertex touches, or you can precompute that value and store it to a second texture.  If your spacing scales much higher, like say 4 or 8 or 16 meters between vertices, though, it becomes difficult to work out the best fitting surface.  For now, I think I will simply max each tier and see how that looks.

--------------------
For 2m spacing, it looks like the best solution is to put each vertex at their regular point, then starting at the top, look for triangles which intersect the virtual geometry, and raise the lower vertices until they are clear. Raising the lower vertex makes the angle of the next triangle away steeper, and therefore less likely to have a collision. I think the same holds for x4, x8, and x16, but I am not entirely sure if that is the optimal way to do it.

You lost me again, but this seems like good news.

Sorry.  I'm trying to figure out how best to fit a low-poly mesh over a high resolution heightmap while also making sure the mesh never crosses below the heightmap.  We can only render blocks if they are below the mesh, so it's very important to get this right. However if the mesh is too loose, it will become very costly to render the blocks, especially along silhouettes.

Ah, that makes more sense, thanks!

Hmm, well here's an interesting idea. Since we care more about the heightfield values being at or below the mesh than we do about the heightfield being globally accurate, when we go to raytrace, we could take the column height as being the lower of the high-res heightfield and the low-res heightfield which generated the mesh. When the mesh is above the heightfield, it will be accurate like normal. If the mesh clips the heightfield, the column height will be shortened to where it follows the contour of the mesh, but still looks blocky.

From a performance standpoint, this might even be better because of how tightly it would fit.

I just tried installing the raytrace shader. It didn't work. I have no idea what's going on, and it's seriously hard to debug.

The 1600th post is always bad news, oh well.

Did you know RAM rated at speeds of 1600 have a higher chance of failing?

Did you know that if you shoot a ram out of a cannon at 1600 m/s it will die?

I made a bunch of mistakes, is why it didn't work, and the multi-tier textures continue to be a problem.   But...

Spoiler:

It's working.  And btw, this area has a resolution of 8m per texel.  That part is working even better than I had hoped.

I don´t know enough to tell if that means any real progress, but the fact that fr0stbyte can post pics now seems to indicate that stuff happens.

Yes, this is progress.  Major progress, in fact, and the results are really encouraging, because it looks like the range we need this tech for is much lower than I thought, meaning better performance.

Spoiler:

Okay, I just realized this is MAJOR progress.
so, after you got that finished, what else do you need to do for the engine to be finished?

Full planets, cubic chunks, network overhaul, cloud framework, flying chunks, modular vehicles, in-world assembly, and a couple odds and ends.
We're not even close to out of the woods, but right now we have momentum, and that is indispensable.

any real-life-work-projects that might take away the momentum approaching or clear coast?

We also need someone to figure out interconnectivity. If only we could get ectrimble's code. Oh well.

Was he really incarcerated? I remember I found his facebook page, but it got taken down/made private when mackeroth sent him a message.

He's dead, as far as we know. We also know he had an epic beard.

Ectrimble's not dead or in jail. I was able to talk to him once, and he's fine but he had a ton on his plate. It's probably not worth trying to salvage what he started anyway. The requirements have changed a lot since then.

But what was his beard status?

That's a very personal question, ACH.

EDIT: Let's not derail another one of poor Frostbyte's threads.

ACH0225 wrote:

But what was his beard status?

Well he had broken up with his girlfriend, so I have to imagine it was majestic.

Probably beer soaked.

Have you ever sampled beer battered beard? Absolutely delightful to the tastes.

So back on topic.
Funny thing happened. I was being lazy this whole time and not releasing my textures when I kill the program. Apparently, it's possible that the driver just won't free up the memory (I played a little Borderlands 2 afterward and I wonder if that was also involved). Anyway, I ran out of memory on my GPU so it refused to load my giant 1m world texture. Since I had the diffuse lighting effect in there for that previous picture, I could still see stuff, but imagine starting up futurecraft like normal and everything looks right, but it is all mysteriously black and white... So weird. Also my PC crashed overnight, so I should probably start freeing that memory from now on.

Oh, also another change of strategy.  You can see it a little bit in the screen shots on the previous page, but there are black walls surrounding each change in resolution on the map.  The reason for this is that the the raytracer isn't able to navigate between the layers, and they won't be able to so long I am using this toroidal access (the wraparound style texture).  Wraparound worked previously because the mesh was defining where the texture coordinates lie on the map, but the shader isn't going to have that context.  It would think the heightmap was valid everywhere because of the wraparound.

Here's what I think I am going to do.  First, for the 1km x 1km range where 1m resolution can exist, have two heightmaps of 512x512 each, or maybe 3 heightmaps of 256x256.  All 3 heightmaps have a UV offset to get the right positioning, but they clamp instead of wrap (if your UV coordinate goes off the edge of the texture, it samples whatever the outermost textel was.  The outermost textel would be a ring of 0.

So, to get the height, you sample all 3 textures, and then sum them together.  The coarsest texture would hold a literal height, possibly multiplied by something to get a larger range, and the finer textures would hold residuals (the difference between the coarse height and actual height).  The advantage of this is that the residual values will always be much smaller so we can express a larger range, and even without the residual you sill have a useful approximation.

======================================

Because the texture is no longer using toroidal access, you need to shift the entire texture as the camera moves around.  If you do this with the shader in a separate program, this isn't too bad, though.

I'm a little hesitant to use multiple textures like this, because we want to keep texture access to a minimum.  However, with caching what it is, the clamping can work to our advantage.  Because out in the distance it is accessing the fine detail textures in the same place each time(along that rim), the results of that sample stay in texture cache.  Essentially, at the farthest distance you are effectively only sampling one texture.  In that case, it is beneficial to use the 3-tier system, because the sampler moves more slowly across the texture space.

=======================================

A third thing to deal with is the ray tracer itself.  What I am using right now is borrowed from Iñigo Quilez's Voxel Edges demo.  The logic is absolutely brilliant, and makes much better use of the shading language than my original plan, but the traversing logic is designed for 3D volumes, meaning it re-samples as it crosses the Y axis, wasting steps.  It also isn't equipped to accept an acceleration structure, so there is a lot of work to do.  Still, it is a starting point, and I am glad for that.