Ah Lua, how do I loathe thee… I mean Love, yeah Love…
Posted by | Posted in Game Development, Lua, Pioneer | Posted on 25-11-2012
I’ve never understood the love given to scripting languages embedded in a game engine.
I’m going to take Lua in Pioneer, or in anything else for that matter but it’s Pioneers that sparked this off. You have a system written in C++, you expose it to Lua with C++ side functions that get presented to Lua scripts, you then program in Lua.
You are still programming, it’s just another programming language. Lua is not King, neither is C++, they’re both just programming languages.
Now, inevitably, the next step occurs: Everything has to be done in Lua.
What was a convenience, or a way of rapid prototyping, or a way of scripting light data handling routines, or for displaying data in a GUI is now doing heavy lifting in the engine at about 1/35th to 1/50th the speed it was being done in the traditionally compiled code.
Of course by this time only experienced programmers can actually write or modify the scripts because to make Lua useful you’ve extended it with home grown libraries & since the purpose of Lua is usually to make designers and non-coders lives easier it has fundamentally failed in this regard by this stage.
Whole systems are exposed from C++ meaning that you’re maintaining code twice except that you’ve exposed the worst bits of C++ via the wooley type unsafe Lua where the most advanced editor has all the sophistication of “Notepad.exe”.
Lua is not king, Lua quickly becomes a ball ache most of the time because it grows out of it’s usefulness, rapidly doubles the amount of work required to maintain engines, and slaughters anywhere it’s used in a performance critical subsystem.
I say this as someone who has programmed using it at several companies and Loves Lua for scripting. I just don’t think it’s anything other than a helper and best if it’s regularly pruned to reduce what it’s used for.
Some things should be moved out of Lua in Pioneer entirely and into some form of structured data generated by a tool. All the LMR stuff is obvious, ship definitions, spacestation configuration info, and ANYTHING to do with vectors/matrices/quaternions.
Other stuff is perfect Lua fodder: missions, trade pricing, defining factions, the GUI and probably a few others.
It’s just so annoying writing something in one language, then everyone wanting it in Lua too. Fuck off. It’s written already. Why have it in yet another language? It’ll be doing the same thing! Only then it’ll be in a language that I can muddle by in compared to C/C++ which I’ve been doing for 18 years (33 now, 15 when I started). What bloody good will that do? Will it mean more people can use it? No. There’s already a load of people who can write in C++ on the project who don’t know/use Lua. If anything it will reduce the number of people who can use it to only those who know/use Lua!
Does anyone really think that something has been done in Lua that couldn’t have been done in the C++ side? No. It does mean however that there’s a shitlod of C++ code, then a shitload of C++ interface code, and then a shitload of Lua code to make the C++ do what would have taken at least one shitload less of interface code to just do directly in C++.
You know what? If you find yourself embedding Lua to make your life easier and to get away from C++ then Lua isn’t the answer.
C# is.
So, is this basically a rant about dynamically typed languages? :) It sounds more like this is an architectural problem than a language issue. It should be fine to have almost all the code in Lua, if there’s a reasonable structure to it.
If all other things are equal, it’s far quicker to develop in something like Lua or Python than C++, and still significantly quicker than in C#. That’s what I’ve seen in recent months when I had to move from Python back to C++, despite having spent about 10x as long learning and using C++. (C# is a great compromise, though. Probably my 3rd favourite language now.)
If you’re finding something’s easier in C++, then chances are (a) you’ve not had enough experience with the dynamic language yet, (b) you’re doing something esoteric that C++ is suited to (eg. bit-flipping, macros), or (c) there’s some problem with your engine that is getting in the way. I suspect the latter in this case!
(Sometimes there’s also (d) you’re trying to do something the C++ way in a language that doesn’t want that. (Like when people type “for i in range(len(collection)): obj=collection[i]” in Python because they associate for-loops with indexing. Instead they should use “for obj in collection”.))
Interfacing between Lua (or Python) and C++ can be basically a one-liner to wrap and expose a function – which is less than the amount of time you waste by using C++ in the first place (due to having to duplicate identifiers across headers and source files), so something definitely sounds wrong on that level.
No no, far from it, hence why I linked to LÖVE (https://love2d.org/) in fact where you can build the whole game in Lua.
Lua and C++ are just the two languages we use in Pioneer rather than this being a Lua vs C++ example. What I mean is that you have you traditional engine written in statically compiled & strongly typed language “XYZ”, but you find this restrictive so you bolt on your dynamic weakly typed language “ABC” and then you start using it…
That’s also actually ok.
It’s what happens next that bothers me and where I think many things are going wrong namely that we start doing lots of intensive maths, rendering setup, and other tightly looped work in Lua. That makes no sense though, in LÖVE it certainly does because everything is in Lua, but in something (for example only) like Pioneer we’ve already got all of that in C++ running 35 to 50 times faster, and we already have the binding overhead, such as it is, so why do it in Lua?
Then there’s the laziness, not that Lua programming is lazy, no it’s the C++ programmers. They think: “I should really move this into data! Data driven design is much better!“, then they take their hard coded table from C++ and paste it into Lua, which they then iterate over in Lua to “Load” the data into C++ … tada, tables in Lua are now “data” because it’s not “compiled” – therefore a designer or artist or sea-monkey can edit it.
That’s not really data, that’s still code, it’s not compiled code but we can all agree that it’s still bloody well is code. So what should have been a tool for creating correctly formatted, checked and verified data, optimised for loading etc, instead becomes a big table of text in a Lua script sometimes with comments about what might be valid values but most often not. This then becomes a case of shit-in-shit-out where bugs arise because no-one knows what the magic hardcoded numbers actually mean without looking in the C++ and hoping that the coder remembered to range check and comment at that end.
Then after all that you have the language evangelists who insist that whatever you’ve written in language “A” should be done in language “B”.
I’ve used A & B because it goes either way.
If I write a tonne of functionality for Pioneer in “A” it’s because that’s my preference. Yes you or I could have written it in “B”, so yes it can be rewritten in “B”, do I want to take it out of the language “A” that I know and convert it all into “B” so that my life is then made harder? No. Will it be massively improved by being in language “B”? No. Will I stop working on the feature if it’s in language “B”? Yes. That’s why I chose language “A”.
I don’t go around peoples Lua scripts telling them they should port it back to C++. Even in the examples above I’m not doing it because Lua is bad, I’m saying it because it shouldn’t be in the Lua either! It should be DATA, not just dynamic code but not code of any form at all. There are a few cases where I’d argue that something should move back into C++ but they’re few and very context specific. Things like repeatedly calling functions in the C++ to do tiny bits of work, mathematical vectors for example, or finding a star system involve lots of little bits of work. That’s really poor to do because you’re crossing over between Lua and C++ a lot, possibly thousands of times. In those cases we on the C++ side should just have written a better interface. So it’s not: “Lua is rubbish“, instead that’s: “My C++ side interface is rubbish and forces the Lua coder into their worst case performance scenario, I have cocked up.“.
Lua, Python, Brainfuck, whatever scripting language you want to use is fine.
C++, Pascal, C#, whatever statically compiled language you want to use is fine.
Mixing the two is also fine, great even, but don’t use your scripting languages as data, and use each of them for their strengths.
None of which seems to have happened at the companies where they’ve used Lua.
That lot is what my rant is about.
Awesome, my clarification rant is longer than my initial rant! :D
Lua is the way – the truth and the light.
So I’m one of the main offenders when it comes to moving Pioneer stuff from C++ to Lua. I don’t have time for much right now, but some random thoughts:
* Vectors and that sort of stuff should not be out in Lua. LMR is the worst example of anything. Its also the only Lua stuff being called regularly. Most frames will not see a call out to Lua.
* Part of the push to move things out to Lua has been the desire for mods to be able to extend the game. There’s two main wins there. One, you don’t need a compiler and associated bits to hack on the the game (a surprisingly high barrier for many, we found) and two, its much easier to compose different mods into a single game (not clear how you’d do that in C++, unless you’re doing some kind of extension/plugin system, but that has its own set of problems and hurdles).
* I’ll accept that we might have moved too much out to Lua, and that some stuff would be better as a focused, structured data format. Ship defs are a good example. The new model system does not have a line of Lua in sight, and won’t. Good tooling is going to help here. That said, the overhead of deconstructing a human-readable data file into memory structures is not too bad if its only done once and makes it easier for that data to be modified or replaced.
* The rule of thumb we try to follow when choosing what to push out to Lua is that Lua should define the “what”, while C++ handles the “how”. “Fly over there” is great. “Turn the ship three degress and fire the right thruster” is not.
Anything else you think shouldn’t be in Lua? You mention ship and spacestation defs, which are in Lua mostly because of their traditional links to LMR (not so long ago they were all in the same code). What else?
Rob, be wary of Andy – he once ate someone’s dog while the owner was standing right there.
That was my dog.
Duncan, I hate you.
I should probably make a post about that incident, it is still the best excuse I’ve ever had for being late to a lecture :)
edit: also, post me my damned house key back! :P