Fr0stbyte's Development Log

Damn, looks like I was wrong. Chunk calculating works correctly. It's running at full when you start because render lists have to be regenerated when you first arrive in an area. What messed me up is that it prioritizes rendering the chunk that are currently in your camera frustum and leaves the chunks behind you for afterward, giving the illusion that it finishes a lot faster than it actually does. Right now, the updater aims for doing 1 chunk update per frame. Trying to lower the frequency to once every other frame would improve the average fps, but would also make the framerate a lot more uneven due to the drastically different draw times between frames.

Looks like, barring putting chunk updating on a separate thread like Optifine does, there's not going to be any quick and easy FPS boost. I mean besides the culling trick. That will still work.

*edit*
No, I wasn't wrong. What the heck is going on? Sometimes even after all the chunks are calculated the count goes back up and is constantly updating. No movement, no changes. There is definitely something happening that probably shouldn't, but no idea what it could be. Frustrating.

*edit*
Apparently that's only on some worlds. Other times there's nothing at all. Maybe there's something going on underground I can't see, like something burning. The updates average between 20 and 60 per second on one of my test worlds. If it is caused by a tick-based event, with ticks going off roughly 20 per second, this would mean there are 3 chunks being updated every tick. The render time graph corroborates that it is tick-based. I wish I could tell which chunks are being updated and why.

===================
It may be a cause of concern after all.
The order that chunks are picked out for updating is determined by proximity to the player. That's why blocks you break yourself are updated the very next frame while explosions and pistons at a distance lag visibly. There's no problem until all the chunks between the player and the unstable chunks are rendered. After that, the unstable chunks get priority every tick, blocking out all the more distant chunks and the ones outside the frustum. With 3 unstable chunks, that's 60 chunk updates, or roughly half of my machine's maximum number of updates per second, so best case 3 chunks are doubling the amount of time it takes to load all the world chunks. The lower the framerate, the larger the impact of these unstable chunks.

*edit*
In fact, you can test it yourself.
In F3 mode, the "E:" on the second line is the number of non-rendered chunks. This consists of empty air chunks and not yet rendered chunks.
Disable Advanced OpenGL, set render distance to Far, put graphics to fast, and look straight up to maximize your framerate. Then watch E as you load a level, or even better, generate a new one from scratch. What should happen normally is that E will decrease steadily, consistent with the number of chunk updates which should match the FPS initially. Then, if you have unstable chunks, E will stop decreasing intermittently or decrease very little and then speed up again, repeating the cycle over and over, until E completely stops and there are still constant chunk updates.

Depending on what's causing it, some maps may have horrible framerate just from too many constant chunk updates. Not sure what, but something needs to be done to limit the amount of resources these updates can take. I guess first I need to figure out what is causing it.

That was getting too long so I'm double posting just this once.

New tidbit, pretty much verifies what I am now calling "tick hijacking".
If you load a world with F3 going and open up the esc menu, you can see E decrease perfectly steadily the whole time, until it stops and chunk updates goes to 0. This is because rendering is still active when the game is paused, but ticks are not. When you un-pause, the chunk update count will rise again. Additionally while you are paused your framerate will be perfectly level on the time graph in the lower left corner. When you un-pause the green (or red) strip is longer on the white tipped bars (helps if you look straight up with advanced opengl off to minimize render time). The white tips are ticks, spaced 1 ever 20th of a second. The extra length of the green/red bars comes from chunk updates, which only appear on the tick.

Now, to figure out what's causing it...

*edit*
There was lava at the first chunk I tried, came up empty on the second. Need a better way to test. What I really need is a way to render only the chunks that have been updated in the past tick.
I might have an idea on how to do that...

*edit*
Found the source the tick hijacking...

This function: World.updateBlocksAndPlayCaveSounds():
This call: func_35463_p(k + rand.nextInt(16), rand.nextInt(128), i1 + rand.nextInt(16));

An unidentified function. Wonderful. Apparently it's called once per stacked chunk (so 16x128x16 this time) per tick. Has something to do with lighting. All the other tick based chunk updates are either plants growing, fire spreading, or liquids expanding.

*edit*
Nevermind. Could be other things, too. I'm gonna stop posting everything I do until I get some more definite answers.

fr0stbyte124 wrote:

That was getting too long so I'm double posting just this once.

New tidbit, pretty much verifies what I am now calling "tick hijacking".
If you load a world with F3 going and open up the esc menu, you can see E decrease perfectly steadily the whole time, until it stops and chunk updates goes to 0. This is because rendering is still active when the game is paused, but ticks are not. When you un-pause, the chunk update count will rise again. Additionally while you are paused your framerate will be perfectly level on the time graph in the lower left corner. When you un-pause the green (or red) strip is longer on the white tipped bars (helps if you look straight up with advanced opengl off to minimize render time). The white tips are ticks, spaced 1 ever 20th of a second. The extra length of the green/red bars comes from chunk updates, which only appear on the tick.

Now, to figure out what's causing it...

*edit*
There was lava at the first chunk I tried, came up empty on the second. Need a better way to test. What I really need is a way to render only the chunks that have been updated in the past tick.
I might have an idea on how to do that...

*edit*
Found the source the tick hijacking...

This function: World.updateBlocksAndPlayCaveSounds():
This call: func_35463_p(k + rand.nextInt(16), rand.nextInt(128), i1 + rand.nextInt(16));

An unidentified function. Wonderful. Apparently it's called once per stacked chunk (so 16x128x16 this time) per tick. Has something to do with lighting. All the other tick based chunk updates are either plants growing, fire spreading, or liquids expanding.

*edit*
Nevermind. Could be other things, too. I'm gonna stop posting everything I do until I get some more definite answers.

World.updateBlocksAndPlayCaveSounds is probably the cave sound algorithm, it checks for 3x3 spaces of air that are absolutely dark and it plays a sound if it finds one.

That's what I thought, too, but apparently this has nothing to do with that. That function does some other stuff besides sound.

Adding these lines to the first:

Block.blocksList[l6].updateTick(this, k4 + k, j6, k5 + i1, rand); //same function as first

World.tickUpdates(boolean flag):
Block.blocksList[k].updateTick(this, nextticklistentry.xCoord, nextticklistentry.yCoord, nextticklistentry.zCoord, rand);

Disabling these three seems to take care of it, though I have no idea what consequences that may have.

For starters, fire doesn't spread or die and water doesn't flow. But on the plus side the framerate is perfectly smooth, so that's good.

*edit*
It's only the last one that seems to do the normal update stuff. The other two I think can update chunks but only do it by randomly guessing. I have no idea what their purpose is.

*edit*
One or both of them are for growing things, which makes sense with the light level checks. And that much is fine. What's not fine is causing a chunk re-render when there is no visual change. Now that I know what at least one of them is for, maybe I can figure out why it's causing an update and whether something can be done about it.

Ran a whole bunch of tests. I think it was the vines. They grow so fast and are so common they spam up the chunk update queue. If you remove them all together the framerate becomes smooth, even with all the update functions enabled. Doing more tests, but I think that was it.

*edit*
Yep. Definitely the vines. There are other things, like flowing water and underground mines catching fire via lava, but the main culprit was vines. That was entirely too much work to figure out, but it did bring up some interesting things about the how chunk updates are handled. Every time you move 16 spaces on Far draw, you have to update 152 chunks of land double whichever ones have water. Considering what they are being used for and the kinds of changes that happen to cause a chunk update, the entire system is shockingly inefficient. For a single block added or removed or a single change to the light level (physical, not time of day), the entire chunk has to be re-rendered. There is no detection for whether the the renderlists are already stored on the GPU and intact, and there are no shortcuts for making single block additions.

If I were redoing the render engine, I would add a tertiary render layer for additions to the chunk, redrawing to that layer whenever there is an addition, and only merging the two when there is a deletion. However, that would be a really drastic measure, and well outside of my intended plans. Might revisit rendering when we start doing planets but until then, I'm done with that bit.

We now return you to your regularly scheduled dev log.

I have a question about the new collision model (how's that going by the way?):
Will moving-block-things using this model theoretically be able to push normal entities (that aren't on top) around without
"grabbing" them? That's one of the few things that irked me about the Zeppelin mod's implementation, that in order to push things, the construct had to superglue them to itself. I realize this has very little relevance with what I've seen Futurecraft is trying to do, but I am nevertheless curious.

Thanks for doing this devlog, by the way, fr0stbyte! I love having an inside look at these kinds of things, and this project is one I'm really excited about, both as a player and a programmer.

Collision model is working well so far, and I'm now focusing on the programming interface to make it easy to implement and expand upon, though I'm limited in what I can test for at this stage.

My intention was to make the play not move with the ship until he makes contact on one of the sides and then will move along with the ship so long as he stays withing the oriented bounding box surrounding the ship. There is no technical need to keep a player "superglued". It's just for the convenience of people flying around.

Ah, ok, that makes sense. I won't pester you about it, 'cause I know the stuff I'm thinking about (large swinging doors and the like) probably won't happen (or need to happen) for a long, long while, till after your part of the project has become its own thing. I'll just sit back until whenever, I guess!

Large swinging doors would actually be pretty easy. I might just make that as an example Copernicus mod.

So, what step of the roadmap would you say your on now? 3?

I was preparing for 2 last week with the rotations and matrix stuff. Now I am firmly on 2. This week's article will be about Separating Axis Theorem and how it can be optimized for rigid bodies of cubes.

fr0stbyte124 wrote:

I was preparing for 2 last week with the rotations a matrix stuff. Now I am firmly on 2. This week's article will be about Separating Axis Theorem and how it can be optimized for rigid bodies of cubes.

WOO! We are that much closer to spaceships! I can't wait untill I see the article.

Fell behind schedule this week. My plan was to set up all the features I might possibly need for the collision engine before moving on, but it's taking longer than I anticipated, even for the basic stuff.
For now, I think I'll code it up as a working first draft, and then go back to add optimizations and special cases as needed. Otherwise I'll be stuck on this step forever.

It won't be a dirty hack or anything. It will just have difficulty with the really big stuff. There comes a point when it's more expensive to do preliminary culling than to just perform standard collision, but if that point still leaves you with thousands or tens of thousands of blocks to check every tick, things are going to get ugly, which is precisely why every cycle counts. One of the annoying things about Java is the lack of fine control you get over how your code is executed, and one of the annoying things in Minecraft is the difficulty in benchmarking performance when there are so many contributing factors which won't show up in a controlled environment.

So still working on the collision engine. If I leave the ship-world collisions out until after pocket worlds, I can probably finish the rest of it this week. Possibly even over the weekend, depending on how much time I get to work on it. Full article about the collision engine will be next week regardless. Then come the pocket world stuff.

...maybe.

---------------------

I'm tempted to start the occlusion culling mod first, as it will more quickly pay dividends, and I need to know how effective it is if I will be relying on it for planet rendering (distant future goal). Though the actually occlusion check is dirt cheap, the original line-of-sight calculations are (relatively) expensive. I was going to update line-of-sight for each chunk on their respective chunk update, but if that's getting bogged down by pointless plant-growing updates like what I discussed previously, then this is only going to make the problem worse.

I found one interesting tidbit, though. Vertex Buffer Objects (VBOs), which are essentially copies of all the draw instructions cached in GPU memory, can be modified on a per-primitive basis without being forced to redraw everything in the compiled list, if I understand correctly. It's possible, then, that you could make a buffer of graphical updates which aren't painted directly to a chunk but instead to a separate renderer which can add or remove elements individually, things like vines growing or grass popping up which don't affect their surrounding blocks. Additions would be trivial, though subtractions would probably require a full chunk update. Still, how often does that happen naturally? The downside comes in that you have to actively maintain and store all those primitives, which is why it's only reasonable for the small-scale stuff. Still, it's a good trade-off.

Additionally, this might be the answer to the lighting problem, specifically, how to cast shadows over a large area without a bunch of jagged and lagging chunk updates every time you move. What you would do is cut out a swath of squares that you predict the ship will pass over in the immediate future and re-render those chunks once nix the squares. Then you put all those squares in a VBO meant specifically for updating the light texture and nothing else. That way you could update the lighting every tick without having to redraw anything more than the affected squares. Even better you could extend this system to override the current lighting engine and provide alternate lighting textures for colored lighting or special effects. That much alone makes this path worth pursuing (eventually, though. Flying ships comes first).

fr0stbyte124 wrote:

Ran a whole bunch of tests. I think it was the vines. They grow so fast and are so common they spam up the chunk update queue. If you remove them all together the framerate becomes smooth, even with all the update functions enabled. Doing more tests, but I think that was it.

*edit*
Yep. Definitely the vines. There are other things, like flowing water and underground mines catching fire via lava, but the main culprit was vines. That was entirely too much work to figure out, but it did bring up some interesting things about the how chunk updates are handled. Every time you move 16 spaces on Far draw, you have to update 152 chunks of land double whichever ones have water. Considering what they are being used for and the kinds of changes that happen to cause a chunk update, the entire system is shockingly inefficient. For a single block added or removed or a single change to the light level (physical, not time of day), the entire chunk has to be re-rendered. There is no detection for whether the the renderlists are already stored on the GPU and intact, and there are no shortcuts for making single block additions.

If I were redoing the render engine, I would add a tertiary render layer for additions to the chunk, redrawing to that layer whenever there is an addition, and only merging the two when there is a deletion. However, that would be a really drastic measure, and well outside of my intended plans. Might revisit rendering when we start doing planets but until then, I'm done with that bit.

We now return you to your regularly scheduled dev log.

Does the mod Optifine do the same thing?
http://www.minecraftforum.net/topic/249637-181-optifine-hd-d-fps-boost-hd-textures/

It does not, though I would love to know what all it is doing. As far as I can tell, it's mostly tweaks improving the utilization of OpenGL and balancing out expensive tasks to smooth lag spikes. My proposed stuff is a lot more aggressive and involves fundamental alterations to the graphics engine. I don't think the two will be mutually exclusive, though.

fr0stbyte124 wrote:

It does not, though I would love to know what all it is doing. As far as I can tell, it's mostly tweaks improving the utilization of OpenGL and balancing out expensive tasks to smooth lag spikes. My proposed stuff is a lot more aggressive and involves fundamental alterations to the graphics engine. I don't think the two will be mutually exclusive, though.

With all this work you are doing on the engine, you could probably release all of this stuff as at least 3 full scale mods.

fr0stbyte124 wrote:

*classified*
*classified*

Wha?

Well damn. I did something dumb and lost my last two days of work. Let this be a lesson to everyone: back up your stuff frequently.
I was going to discuss collision today, but since I am now thoroughly behind schedule, I'm gonna keep on working.

To stem you over, I made a quick video of what I got done last week:


"Spinning box, what's the big deal?" You say. Well, it just so happens that this particular spinning box is doing so in world-space while utilizing that quaternion-based rotation system I outlined last time. So not only can it rotate about any conceivable axis, it can also smoothly transition to and from any orientation. It doesn't look like much, I'll admit, but it's an important step.

Collision article is definitely coming once I've caught back up. Maybe Tuesday?


Also, @Buggy
They're classified because they won't be part of FutureCraft and I don't mean to distract. It will be cool though.

Shame about the losing 2 days of work thing, but at lwast you know at least some of what you lost so you just have to type back everything you rewrote.


And wow, the quaternion rotation is REALLY smooth compared to all other rotations i'v seen.


And a crazy idea for something you could add to the future plans area, what about quaternion based three dimensional rotation for a player. So when a player is in gravity they are upright and can walk, but when in zero g they float around and can be bumped around by things hitting them.

Nice rotation.

Wow, that sounds really weird...
Especially if I mentioned it off topic; just mention at some random time:

"Hey, nice rotation!"

I tend to agree with Buggy, the rotation is a lot smoother, faster, and no doubt less memory intensive than the rotation engine use in the Zeppelin Mod. Primitives.

Dr. Mackeroth wrote:

Nice rotation.

Wow, that sounds really weird...
Especially if I mentioned it off topic; just mention at some random time:

"Hey, nice rotation!"

I tend to agree with Buggy, the rotation is a lot smoother, faster, and no doubt less memory intensive than the rotation engine use in the Zeppelin Mod. Primitives.

The real test of the new model will be with the ship-world and ship-ship collision. It should be a lot more efficient than Zeppelin, but ships (and other moving bodies) will potentially be much, much larger. It's not impossible to have scenarios in which thousands of points need to be transformed into a different world space in the course of a single tick. Won't that be fun.

fr0stbyte124 wrote:

Dr. Mackeroth wrote:

Nice rotation.

Wow, that sounds really weird...
Especially if I mentioned it off topic; just mention at some random time:

"Hey, nice rotation!"

I tend to agree with Buggy, the rotation is a lot smoother, faster, and no doubt less memory intensive than the rotation engine use in the Zeppelin Mod. Primitives.

The real test of the new model will be with the ship-world and ship-ship collision. It should be a lot more efficient than Zeppelin, but ships (and other moving bodies) will potentially be much, much larger. It's not impossible to have scenarios in which thousands of points need to be transformed into a different world space in the course of a single tick. Won't that be fun.

Lets hope that it won't be too bad, or if it is too bad then its not horrible.

Also, seeing as afaik your going slightly out of order with your development list and a few things changes since you made it, I decided to rearrange it slightly so its more accurate to the route your taking.

1: Render
2: New Collision Engine
3: Isolated Pocket Spaces
4: Render Revisited
5: Object Interaction (redstone, construction, etc.)
6: Dynamic Lighting
7: Space...stuff
8: The stuff under Future Plans.

Some of it might be wrong, but as far as I can tell your working on S-W and S-S collision right now and just finished up on the new collision engine, so I combined them. This one is slightly iffy, but I placed Isolated Pocket Spaces where the Pocket World was, because as far as I know dimensions are relativily easy to add, and it makes more sense to mess with isolated pockets while your allready messing with that section so theres the possiblity that you might do that. I left Render Revisited in there though, because tbh im not sure what that step is about

Nah, I'm still leaving S-W and S-S collision for a later day. For one thing, moving ships spawn on top of the original. It would be problematic if terrain collisions were already in place, so pocket worlds need to come first.
If it turns out simple enough, sure I'll merge the two pocket world steps.

so how goes the development? i was wondering how the game handles it when you fly very fast and the chunks stop loading fast enough for you to see the ground? does the game let you keep flying?

The development goes slower than I'd like, but that is my own disorganized fault. In the later versions of the mod there will be a shiny new variable detail engine, which will allow you to move around much more quickly and see further by generating terrain of lesser detail faster. The trick will be making it look not crappy and is the subject of ongoing research.