Writing a Game Engine

[sourmìlk]

As some of you may have heard, I'm writing a game engine. This is just as a hobby, and there are no time constraints. Nevertheless, it's a kind of large project, and one where I'm learning things as I go along. This may be kind of a stupid question, but does anybody have some general advice / suggested reading?

(The game engine will be in C++11 using OpenGL 4.2)

[Divinas]

Some non-coding general advice:
Make some kind of schedule: which features you want when. Do things incrementally: as in, get something working that renders something on screen as fast as possible, and start extending it. Have a specific game in mind when you design your engine. Most modern engines are pretty much game-independent, but they have become such through much abstraction and understanding of what is in common between all games. But you'll have a much easier time if you know exactly what you want to accomplish. Try to work on both gameplay, physics and graphics at pretty much the same time. This is not how stuff is done in commercial projects, but (from my own experience) it's much easier when doing something by yourself: it's hard to imagine what you might need when you're just doing graphics, and how you might want to use that thing (especially if you have no experience with other engines)

[Yakk]

Go here, and read and learn:
http://www.gamedev.net/

[BombSite_A]

My friend and I are writing a game (well, he's doing the resources and I'm doing code and design). We were going to enter it into the Dream Build Play contest, but we decided we didn't have enough time to finish and we were both technically too young. I am only 5 days to young.
Anyhow, I would suggest you get a class ready to handle drawing sprites. This will make your job a lot easier. Make everything very object oriented so you just need to instantiate the class if you need anything. If this doesn't seem obvious, don't make a game; just learn how to program instead.
I would also find a good API for making games. In C# I use XNA, I don't know what you use for C++. Don't reinvent the wheel!

[sourmìlk]

I like reinventing the wheel. It helps me understand the wheel better.

Also, I have a question: should I use an entity-component system for the entire engine or just for the game logic layer? Or should the 3D rendering layer also be entity-component?

[Divinas]

It should definitely be entity/component based. But... make sure you know what exactly that means. Most stuff that is renderable on the screen should be an entity - be it a particle system, ordinary mesh, some procedural generated thing. But your UI is most probably not one of those.
Your renderer itself though, is neither an entity nor a component. It will most probably be some singleton somewhere, that reads data from your entities and renders stuff on screen. That can be mesh data, if your entity has some kind of mesh on it, or particle state if it's a particle emitter, or procedural geometry, or whatevers. Your shaders should be some kind of data, that is neither an entity, nor a component, but it should be a property that you can edit on your meshes component, or can be added to your procedural geometry, or your particles.

[sourmìlk]

Okay, that makes sense.

[freakish777]

Your renderer itself ... will most probably be some singleton somewhere

You know what would be interesting? A game which scales with the number of screens the user has (or, more appropriately, scales with the number of windows they want open for the game). Perhaps it gives them another view for each window? Imagine a multiplayer game (like LOL maybe?) which lets you view what your friends are up to in a second window.

Obv, controlling things across multiple windows wouldn't really be feasible... but displaying multiple views is easy enough and could add some strategic elements to a game which haven't really been explored yet.

[Divinas]

Well, separate render targets don't mean that your renderer shouldn't be a singleton somewhere. You most probably already render to multiple render targets either way.

p.s. Small correction, i thought that the upper post was by the OP.

[Sc4Freak]

Although unless it needs to be a singleton, don't make it a singleton. The only time you'd use a singleton is if it would be logically incorrect to have more than once instance. That doesn't apply to a renderer, so there's no benefit to making it a singleton.

I like reinventing the wheel. It helps me understand the wheel better.

Also, I have a question: should I use an entity-component system for the entire engine or just for the game logic layer? Or should the 3D rendering layer also be entity-component?

The "entity" in "entity system" refers to in-game entities, not abstract program objects like a Renderer or resource manager or whatever.

Gmes should be thought of as simulations. You have some virtual world that you're simulating, and the player is interacting with that virtual world. It doesn't matter if it's an RPG, RTS, FPS, or whatever - the principle is the same. Inside this virtual world you have entities - a car, an enemy, a weapon, an explosion, etc. These are all objects that exist inside your virtual world, inside your simulation.

Those entities are what you use entity systems for. Is a "renderer" some object that exists inside your virtual world? Nope. How about the UI? Nope. Windowing and input managers? Nope. You can use traditional OO for these objects.

Most of this seems like common sense but it's actually something that a lot of people miss, I've noticed.

[Sc4Freak]

Oh, and I forgot one thing that I was going to say earlier:

Making a game engine on its own is actually not such a good idea. Game engines exist for only one purpose: to make games work. Constructing a game engine in a vacuum just plain doesn't work. You end up with a giant mess of very abstract, generic, reusable, and elegant code that's completely useless for game development. The best way to build a game engine is to build a game with an engine, then pull out the reusable parts into a standalone engine when you're done. That way the development of your engine is guided by the actual needs of building a game - and when you're done, you can point to your game as proof that your engine actually does work in practice.

tl;dr: make games, not engines.

[sourmìlk]

Most of this seems like common sense but it's actually something that a lot of people miss, I've noticed.

Well that's what I intuitively thought, but I wasn't sure.

Also, I suppose I could work towards a specific game and add reusable components to the engine as things go on.

[Divinas]

Oh, and I forgot one thing that I was going to say earlier:

Making a game engine on its own is actually not such a good idea. Game engines exist for only one purpose: to make games work. Constructing a game engine in a vacuum just plain doesn't work. You end up with a giant mess of very abstract, generic, reusable, and elegant code that's completely useless for game development. The best way to build a game engine is to build a game with an engine, then pull out the reusable parts into a standalone engine when you're done. That way the development of your engine is guided by the actual needs of building a game - and when you're done, you can point to your game as proof that your engine actually does work in practice.

tl;dr: make games, not engines.

Very much this. I have the weird feeling I said this in some other thread, but yeah - this. Doing abstract stuff is mostly useless for several reasons.
1. You don't have anything to show or anything complete.
2. Unless you're very experienced, you most probably have no idea what you need, or how to organize the thing that you know that you need.
Just think of some game - it shouldn't be too simple, but it shouldn't be too complicated, either. Think of some features that you want in it, what would those features require.

If you want to build an engine that can handle some of the most important (or more correctly said, commercially popular, those are things that practically every engine has) things up to date, here's a (very in-comprehensive) list of 'systems' your engine should have:
Mesh rendering, particle rendering, a material editor, resource manager, rigid body physics, collision detection, soft body simulations, animation, cutscene scripting, terrain analysis for navmesh generation, UI solution, serialization manager, some kind of reflection system, raycasting, shapecasting, shadows, post processing fx, control schemes, camera editor, procedural animation system.. and countless more that don't come to mind. Trying to implement all of those things from scratch, for a single person, is a herculean task, and one that you most probably can't accomplish unless you're very very stubborn (or motivated ).

[sourmìlk]

I'm definitely on the stubborn side. And I have an indefinite amount of time to implement these things

[Inglonias]

I'm definitely on the stubborn side. And I have an indefinite amount of time to implement these things

Time != motivation. If you aren't working towards an end project, you are far more likely not to finish all of these things.

I never really have time constraints on projects, and not finishing anything is the biggest vice that I face. The only projects I have truly finished are those which I am constrained by time for. (Read: School assignments.)

Motivation is key, not lack of time.

(Note that I've never coded a game engine either, so YMMV)

[sourmìlk]

I get that, I wasn't trying to draw an equivalence. I think part of the reason you complete time-constrained tasks is that the constraints are the only good definitions of when you've actually finished.

[Divinas]

You know best, I'm still advocating to try to do something specific with your engine, instead of just abstract code that does nothing. Commercial engines have tens or even hundreds of thousands of man hours of work put in them, you're obviously not making something like that, but still, if you want to have everything, that's rather impossible and demotivating. Have a specific game in mind when doing your engine. If you finish it and still want to extend it ,think of some other game and start work on it.

[WarDaft]

If you want to build an engine that can handle some of the most important (or more correctly said, commercially popular, those are things that practically every engine has) things up to date, here's a (very in-comprehensive) list of 'systems' your engine should have:
Mesh rendering, particle rendering, a material editor, resource manager, rigid body physics, collision detection, soft body simulations, animation, cutscene scripting, terrain analysis for navmesh generation, UI solution, serialization manager, some kind of reflection system, raycasting, shapecasting, shadows, post processing fx, control schemes, camera editor, procedural animation system.. and countless more that don't come to mind. Trying to implement all of those things from scratch, for a single person, is a herculean task, and one that you most probably can't accomplish unless you're very very stubborn (or motivated ).

However, almost all of those things need to know absolutely nothing of the inner working, or often even existence, of any of the others. Rigid body physics doesn't change based on whether or not you have particle effects, and vice versa. It is at least theoretically reasonable to consider a game engine which contains none of these things, but has an interface capable of connecting most of them with little effort, and some with more (or a lot more) effort. Implementing all of that really is too much for one person as a side project, unless you can cut it into tiny chunks each of which can be its own side project - functional in and of itself.

If you do want to just write a game engine, I suggest starting with a completely abstract world representation, no ability to display things at all, interactions classified only at the most general level (so you can substitute in collision detection in roughly the same amount of effort as fluid simulation) to keep track of "what's what, and what it's doing" without really specifying any particular way that things actually change, or look. Basically, only a step removed from object oriented coding itself. Then build a basic way to display things, without changing your abstract world in any way. Then start adding possible kinds of interaction piece by piece. Make sure each is fully functional without being plugged into a world at all. Add better ways to visualize things piece by piece as well. Once you find that interesting things can happen, add a scripting language, make it as general as possible. Then, extend your scripting language to allow the creation of a user interface. From there, just keep adding the next thing that seems interesting, incorporating it into your scripting language as you go.
This will be a lot harder than doing it with a particular goal in mind before you start, but if that doesn't bother you... well, happy hunting!

[sourmìlk]

I don't know best, Divinas

WarDraft, I think you're underestimating the size of the displaying of things. That's actually tens of thousands of lines of code, at least.

Sc4Freak, now that I think about it, I'm actually kind of surprised that you think it's okay to write the 3D rendering stuff in a normal OOP patter, and then use those objects to build components and entities, seeing as you said that all components should be PODS.

[Sc4Freak]

The two concepts are orthogonal. Using OOP patterns for your renderer or sound engine or whatever doesn't mandate a particular design for your entity system. For example, you might have some components that handle various aspects of rendering and transformation. If, for example, you use a traditional OOP design for your renderer, you can simply have your component managers take the data in the components and submit it to the renderer. It doesn't affect the design of your components themselves.

[sourmìlk]

Well the components might have to hold 3D-rendering objects, making them not PODS, right? Like a model / drawable component or something could should have a VertexBuffer member, and if VertexBuffer is an object, that makes it not a POD, right? Or am I still going about thinking about this the totally wrong way? (I'm not as good at composition as I'd like to be).

[Divinas]

Actually, what Sc4Freak is saying, and I'm completely agreeing with him when talking about the renderer(unless I misunderstood him), is to have no logic inside the renderable components.
You have an entity, which gives you a world position. It has a visual component, as it is customary to call those things. This visual specifies the shaders you're going to use, the geometry, any additional data you might have to use (like vertex winding, hiding the visual for some gamplay needs, whatever else you might need). The visual has no logic inside it - it is just data, maybe with some callbacks to ensure consistency and that internal structure is maintained, but it doesn't know that it is a visual in any way. Your renderer gets all your entities, gets all visual components, paralelizes your rendering jobs, and starts fetching geometry and rendering. In this sense, your visual component is POD. It might not be under the strictest of definition, but it definitely doesn't 'do' anything on its own.
A small addition, Usually visual components hold references to meshes, so they are seperated and you can reuse different meshes with different settings, so that they can be referenced and you don't have a clone of the vertex data for all the entities that are copies of each other, but this is a bit of details. Not every object in your is a component or entity, even though it's data!

@WarDraft: That's exactly what I had in mind. There are a lot of things in an engine, and trying to build those things without any goal is very very hard. If he has a specific goal, he'll build the only parts and systems that he needs for his specific game, and will see how those systems are used, which will be much better than just writing code that does nothing