Times, trials, and turbulence.
Tonight’s testing session was a most illuminating one. I managed to get up to five players on total at once, which is (surprisingly) the most that the game has seen at this point. The networking engine seemed to handle the multitude of players pretty well, and there no were visible lag spikes. We had a couple of overseas players that suffered from a fairly unavoidable latency addition.
(As always, massive thanks to tonight’s host of testers (Knarkles, Rip-Off, Prinz Eugn, and Danger))
The lesson: following the rules can be “bad”.
The session was going pretty well overall, when some “strange” things happened. Really peculiar things like players sometimes being unable to throw their weapons away, or players claiming items were on the ground in spots that the server saw nothing at, and the ilk. In a nutshell: really weird synchronization bugs.
As you may know, Skirmish has a lot of logic dedicated to propagating changes in props across the network to other players in an attempt to keep the game simulation looking and behaving the same on all of the clients’ computers. The problem is herein: the server sends out messages to clients dictating what events are to transpire. However, there is a fair bit of client-side logic that prevents seemingly impossible things from happening.
For example, there is a restriction that the player can only have a certain number of certain items in his possession at once. Such as only 1 Assault Rifle at a time. What happens if a client has 1 of these items, but receives a message from the server saying to add another to their inventory? Huh? What does the client do? Allow this should-be impossible action to occur anyways (on the basis of, “the server said so”), or stop it outright, knowing that it’s just not sensible?
The current logic is set to work such that client-side logic wins the battle in the case of an argument between server commands and illogical actions. The problem is, this causes some pretty nasty synchronization problems when the server tells the clients to do something that is (from the server’s perspective) totally logical, but the state of the clients says that it’s just plain silly. Normally it’s not a problem, but even tiny slips in client-side logic can lead to all of the nasty bugs that I mentioned earlier.
All of that said, the lesson is that there needs to always be a very clear course of action for any significant event that can occur. And when there isn’t, the client needs to keep a verbose log of whenever local client-side logic prevents a server command from being carried out. For anyone else embarking in any online game projects, I beg you to keep this in mind. If you don’t, some of the nastiest to track down bugs will rear their ugly heads. 🙂
Artistic talent climbs aboard!
By some chance, I happened to be low on testers and invited him into a short Skirmish testing session. Nothing fancy, certainly, but soon after he expressed his interest in doing some sprite-work for Skirmish. He’s not contracted or anything to the project, but has agreed to do some art (specifically, firearms and player sprites) on a as-he-has-time basis.
I’ve recently plugged some of his revamps of the existing weapons into the game, and I’m liking them a whole lot. It’s hard to tell when you have sprites so small, but overall just the look and feel go a long way in making things look more professional. I was practically oozing with anticipation when he mentioned the reloading animation he was working on.
Anyways, a huge kudos to Mark for his interest in Skirmish and the stuff he’s working on. I’m looking very forward to plugging more of his stuff in! 🙂
A Friendly Testing Session
Managing testing sessions for an online game is something that I don’t think I’ve ever really spoken about, but there are certainly words that need to be said on the topic, for those of you who haven’t had the experience. 🙂
Testing is a task with a difficulty exponential relative to the number of testers involved. Now don’t get me wrong — all of the folks that help me test Skirmish are absolutely awesome folks — but the management of it all is rough. Tonight only saw two other testers, so it was very manageable, but as the numbers increase (try 10+!), things get challenging.
The first step is to alert everyone about the test. This can be done via long-term notice (ie. email) or short-term notice (ie. instant messaging). I always recommend the former when it’s possible, but oftentimes the case is simply that a feature has just been implemented, and (desperately!) needs testing. This consists of messaging several people — as many as you dare 🙂 — and explaining the situation to each of them. Then comes launching the game and meeting everyone inside the game itself. If anyone has any problems launching the game, or additional questions, or can’t come on for another 5 minutes, things get held up for everyone. And from the perspective of a tester, well, it’s not that much fun to sit around waiting on some guy who was pleading with you a minute ago to help test.
Once the monumental task of getting everyone in-game has been completed, trying to get a general direction going for the testing is the next step. Answering questions, introducing testers to other testers, and telling people not to play with the rocket launchers all occur frequently, so you need to pay attention to everyone AND try to keep track of bugs and events that transpire within the game. And if anyone does have a bug or glitch, you need to either plead with everyone to stop testing, or try and address that tester’s bug while hopefully not missing others. It’s a mad-house when you get over 5 or 6 testers at once.
And that’s it. There’s no real moral or lesson in that; unfortunately it has just degenerated to a rant about how testing can be painful sometimes while developing an online game. A sort of “dark side” that I don’t think a lot of developers really think about beforehand. 🙂
Tossing the ‘ol Pigskin Around
On an opposite note, tonight’s testing was quite enjoyable and illuminating. Big thanks go out to Steve-o and Cobrask for their invaluable help. Some bugs were uncovered in the process, and also revealed that my movement prediction (again!) needs some more work. It starts to break down and look ‘jumpy’ once player pings move over 150ms or so.
I took a small handful of screenshots while we tested, which I shall affix tag-lines to. 🙂
(Getting everyone settled into the one-room map.)
(Tossing weapons into a big heap in the centre of the room. “End the violence!”)
(Finally, things just got plain out-of-hand. Had weapon firing been implemented, it would have been a blood bath.)
The focus of the testing session was to see how well movement prediction stood up against higher pings, and to test the two most basic player actions: picking up items, and throwing them about. The latter worked quite well, and I’m feeling ready to move on to (finally!) implementing weapon firing and reloading, which should be exciting territory indeed. This weekend is a long one for us Canadians (Canada Day), so I’m hoping to maximize the progress that Skirmish will see.
Working on the “Inside”
As some of you may know, at my university we alternate every 4 months between studies and co-op. Near the beginning of each term of studies, we apply to and have interviews with employers via an online job application system at our uni called JobMine.
Anyways, I’ve been through seven interviews over the last month with various companies, and today was the point where I got to see which companies extended offers of employment to me, and was to make my choice of which offer to accept. Most were software development companies, but I instead decided to take a fairly off-the-wall job compared to what some might expect: tutoring at the university.
I will be tutoring a second-year Computer Science course (which happens to be my favourite CS course that I’ve taken so far (involving assembly, parsing, and basic compiler creation)), which will involve giving weekly tutorials, marking assignments, creating model solutions, proctoring/marking midterms and exams, and holding office hours (and likely some other miscellaneous administrivia). In terms of work-load, it’s pretty light (35 hours/week, and very close to home), so I’m planning to take a course at the university itself during the term, and perhaps even freeing up more time for Skirmish. 🙂
Skirmish: Player Actions
I’m feeling comfortable with where the prop simulation work I’ve been doing is, such that it’s now at the point where props can be seamlessly created, flung about, moved in/out of players, and destroyed, all over the network without any noticeable glitches. Now comes the implementation of allowing players to interact with this system of prop simulation: via player actions.
The atomic actions that a player will be able to execute are both few and simple: start/end use, start/end throw, reload, interact. “Use” means to perform the primary use of the currently-held item, which may be firing a projectile, healing a teammate, or detonating a bomb. Throwing and reloading should be straight-forward. Interact is the action of attempting to ‘use’ a prop that is on the map near the player. Interacting with an item results in picking it up, interacting with a switch might cause a door to open/close, interacting with a vehicle might cause the player to enter/exit it, etcetera.
After the completion of player actions (and a healthy amount of testing), the fundamentals to actually having a somewhat playable ‘game’ will be in place. Players will be able to interact with the environment and fire weapons at other players. Player death and respawning will come soon after, and Skirmish will be roughly at the same stage it was at in its previous iteration, albeit far more stable and scalable. Exciting times ahead. 🙂
I think I’m finally at the point where I can call the movement prediction — that’s the logic to perform both ‘guessing’ of where players will be based on old data packets, and the correction logic when prediction errors occur — as being in working order. I haven’t tested it much under high-latency situations, so more fine-tuning will definitely be needed in the future, but it’s at the point where remote players generally appear to move around smoothly and without visible delay.
How does it work? The first half is in performing good estimates of where a player will be after a certain amount of time. Player movement information is sent at a default rate of 4 times per-second. This player movement data packet contains the player’s location, velocity, rotation, and the keys that the player was pressing at the time. From this, once the packet arrives, a decent estimate of how old the contained data is is performed (for now, simply adding client1-server and client2-server latency together). Given a good guess at how old the data is, the remote player’s position/velocity/etc is extrapolated using the regular local player updating logic to where that player should be.
The next step is applying smooth corrections. Usually players will move in pretty predictable ways (eg. in a straight line down a long corridor), so this isn’t often a problem, but in complicated gun fights or when pulling off tight dodges — especially with players that have higher latencies — estimates can quickly diverge from what is ‘expected’. Typically the error isn’t too large, so the real problem is just to move the player from his previously estimated spot (now incorrect) over to the newly-received spot (which is correct) without the user behind the monitor noticing the discrepancy. In previous projects I would just apply linear interpolation between the current-position and the newest-position, which technically worked, but looked pretty cheesy (and obvious) to anyone looking at it. I instead opted for a cubic spline, which is composed of a start point, an end point, and a control point. This creates a smooth curve whose direction is influenced by the given control point. By using a start point of the old position, an end point of the newly-received position, and a control point of where the player would be in the future if the new position didn’t arrive, it creates the illusion of the player continuing to move in the same direction, but smoothing deviating to the player’s true position in a non-obvious manner.
This is the next area I’ve been working on, which is also the last stepping stone before Skirmish is essentially playable as a game, rather than just a neat little demo. 🙂
Props are the most generic object in the game world. Crates, desks, chairs, rockets, firearms, grenades, and shrapnel are all props in Skirmish. A prop could be defined as a “game object that is physically interactable and simulated. Most of these props need to be synchronized such that they behave the same on every player’s computer, which is the crux of the problem.
The current development item is to create a manager which will automatically track the state of the simulation of the players, and send out network messages to update props as appropriate, such as when a player picks something up, throws it away, uses it, or, say, steps on a mine. Doh!
I’ll be talking about this in more detail once I have it implemented and in proper working order. So, here’s to looking forward to smashing some crates together. 🙂
Skirmish Breaths, Slowly
It’s been very busy here lately, with a combination of assignments and the co-op process through my university. That means resume-building, applying for jobs, and, of course, interviews. My schedule is full of them in the next couple of weeks, so progress on Skirmish will naturally suffer some slow-down.
The current development item has been movement prediction, which is actually a very broad topic in network games/simulations: the ‘art’ of guessing where a player/object will go ahead of time (since all data received via the network is delayed by some amount of latency), and seamlessly applying corrections when the predictions are not entirely correct. This is very difficult to do well, and it is critical to get it done correctly. Poor movement prediction is evident in many online shooter games, when you find that shooting directly at an enemy has no effect, or when you need to ‘lead’ your enemy (ie. fire ahead of where he is moving) in order to land hits. If even many commercial online games suffer from this, rest assured that it will certainly be a challenge. 🙂
Free plug: onGameDev.com
I wanted to offer a plug for the ‘underdog’ game development portal, onGameDev. They’ve been doing some revamping of their layout and structuring, and are looking to getting off to a really good start. The main shortage is in their supply of members, so you’re encouraged to pop by and check out the fairly close-knit community, and perhaps even join in on the party.
…And it was good!
During the weekend, Skirmish saw a storm of coding. I hammered away at the game networking system as quickly as I dared, tasting that glimmer of online play that laid only over the horizon. Or more simply put: just really darn close to being playable online.
Below are a couple of screenshots to summarize the end-result of this work:
(Huge thanks go out to Dean and Brian for helping pioneer online play.)
The bottom line: Skirmish is finally ‘playable’ online! It’s only the bare bare basics of the game networking, but after all of these months of work, it’s finally all starting to come together. 🙂
What you can do: Host games. Join existing games. Move around and see other players join/exit. Send/receive public chat messages.
What is not (yet) implemented: Synchronization of props and player actions across the network (encompasses static props, items (guns, etc), and firing of weapons). Player health and death. Movement prediction. And heaps more. 🙂
The important thing is that Skirmish is now at a turning point. No longer is it “that game I’ve been keeping to myself on my local machine”, but “that nifty new online game that folks can actually sort-of play”! I’m hoping that in having even just this much available for public consumption will help: a) get bugs reported quickly, and b) bring more public interest to the game, now that I’ve established some more credibility to the name of the game.
Naturally, the development of an online game tends to require multiple users in order for many facets of the game to be adequately tested. I still follow my tendency to randomly yank folks off of MSN when I spontaneously require some testing, so if you’d like to take part in this early stage of alpha testing, please (please!) add me to your MSN (hopedagger at gmail dotcom) and let me know you’re along for the ride. Tests will usually be fairly short in duration, and, of course, testers will be credited for their invaluable help.
Lastly, I’d like to once again extend a massive thanks out to everyone who has been a supporter of Skirmish. (Yes, I tend to get a little misty-eyed every time Skirmish gets to this stage. 🙂 )
Up and Running
After an unreasonable amount of time and effort, I managed to recompile and set up the Master Server on my server box. The router I’m using (thankfully) has a setting to automatically use a dyndns.org account to update with my latest IP, which means that my residential dynamic IP won’t be an issue in long-term hosting of the Master Server.
Milestone 4 Client Available
With Milestone 4 comfortably completed, I’m eager to finally put up a link here so I can get some real feedback on the interface to the Master Server, as well as catching any bugs that are still lingering about. Or just reserve your favourite character name. 😛
Recall that the deployment of the game requires Java 5, and either Windows, Mac OSX, or Linux.
Also worth noting is that the Master Server is still in its infancy of long-term testing. It may go down periodically while the kinks are still being hammered out.
- Create/log-into an account
- Create and delete characters
- Configure user settings (key bindings, interface options, etc.)
You cannot yet:
- Host games
- Join games
It’s true that you can’t do any “playing” yet, but enough infrastructure is now in place that adding these two components are not major additions. Hosting is actually already in place, but removed in this release simply because there’s no real point in it until the game networking is implemented. One might think of this as a, “now all of the boring work is out of the way” release. 😉
There might not be a whole plethora of things to test, but I’d be very grateful for any feedback, comments, questions and suggestions that come to mind while looking at it. As usual, a huge thanks to everyone for the support and encouragement along the way of Skirmish’s lengthy development. 🙂
Frequency of Updates
It’s been pretty quiet around here for the last few weeks, but I’m still hoping to increase the frequency of my updates as of now. I’m becoming more or less adjusted to being back in classes, and have my weekly course-load pretty comfortably gauged. 🙂
It brings me plenty of pleasure to say that the initial version of the Master Server is ready to roll. Finally both the Master Server app itself and the game client’s interface to the MS is in good enough shape to be warranted as “done” for the time being. There are still more features and additions (on a more minor scale) to come in the future, but all of the core functionality is solidly implemented. Account creation, logging-in, character management, game hosting, receiving game listings, and user configurations are all in working order.
The Next Step Toward Being “Playable”
The only real obstacle left is getting the Master Server up and running 24/7 on my in-house server. Since we’re using a pretty standard residential ISP, the option of a static IP isn’t feasible. What this means is that the IP address of the machine will change every so often (although it hasn’t in over 2 weeks so far), which would render any hard-coded IP in the client worthless. I’ll be putting it up anyways soon for the sake of having it up, though.
Once the Master Server is up, I’ll release the latest version of the client (the completed Milestone 4) and put up a Webstart link. Technically, users will only be able to create accounts, log-in, and perform character/settings management, but it certainly will be something tangible.
Milestone 4 encompassed the development of the first version of the Master Server, and the first iteration of the game client’s interface to the Master Server. Now that I’m comfortable with where it stands, Milestone 5 can begin. It will focus on arguably the most crucial component of all in Project Skirmish: the game networking. Development will focus on writing the server and client for the hosted games themselves, and more actual game logic to support real games.
This entrenches upon areas like movement prediction for players, managing concurrent physics-simulated Props in the game world, player health and deaths, adding more weapons/items/content, and finally: Alpha testing. I’ll be posting around some websites asking for some testers, if enough of the local folks I already know here aren’t interested.
I’m planning on talking a bit about the networking model I’ll be using for Skirmish, which is largely inspired by the Quake 3 networking model. Until next time. 🙂
Things have been particularly hectic lately, with my co-op term coming to a close, classes starting tomorrow, and the lengthy work-term report that I’ve spent the weekend frantically composing. Things should balance themselves out in the coming week or so, and Skirmish development will resume once I get a solid hold on my upcoming homework load.
Since my last update, I’ve finished the User Settings/Configuration set of menus and windows. This means that the only item left before actually game networking (ie. the game actually starts to become “playable” online :)) is the game host tracking on the Master Server’s end. Exciting waters lie ahead!
A little while back, I briefly spoke about my entry in my university’s gamedev club’s 24-hour competition. All three judges have finally posted their marks and opinions of the entries, which you can view here, as well as the entries themselves. A huge congratulations to everyone who took part — it was a whirlwind that I can wait to help organize again this summer. 🙂
For anyone looking for a quick link to my entry, Circuit Runner, you can download it at: Circuit Runner.