part 1. part 2. part 3. part 4.
Since the first video games gameplay elements and actions were inspired and modeled after real life. Despite layers of abstractions and processing limitations we used whatever input devices available to bridge the gap between the virtual world and ours. To help in this process we designed mechanics to be intuitive so that our lived experiences would help us better understand the game. Now, over three decades later, the industry now supports two controller technologies that allow players to move and play intuitively; motion controls and camera based controls.
Play Like A Pro
Motion control is a term that applies to devices that use at least one of the following technologies designed to sense motion and position: linear accelerometers, gyroscopic sensors, rotation sensors, rate sensors, and magnetometers. To keep things focused, I won't go into the technical side of these parts.
Both the Wiimote+ (Wii remote with Wii motion plus functionality) and the Playstation Move controller work with their pointer features to most accurately calculate where the controller is in and how it moves through 3D space. Without the pointer technology, both devices are a combination of about 7 sensors or so of various types. In the same way that the D-pad is a combination of 4 digital buttons, the Wiimote+ is a combination of several sensors set in a specific formation to sense motion in any direction. With controllers relaying information back to the system faster than 1/60th of a second there is a high level of sensitivity and precision possible from motion sensing controllers.
Yes, the Wii and the Playstation Move are the most obvious examples, but motion controls have had a much longer history especially with handheld gaming.
- Kirby Tilt 'n' Tumble (see video).Using a series of accelerometers, tilting the Gameboy tilts the game world maneuvering the ball like Kirby. Popping the Gameboy upward caused Kirby to jump. Before playing the game needed to be calibrated on a flat surface.
- WarioWare Twisted (see video).The WarioWare games are expertly designed. The entire concept of micro games that accelerate and challengethe player with mixups is solid. But beyond this, each game in the series is designed to take advantage of a different type of technology. With Twisted, the game features a tilt pack powered by a gyroscope designed to detect angular movement. Calibration is done after starting up the game and more subtly after each microgame.
- Wii.Holding a TV remote like motion controller frees the hand and range of motion in ways that were not possible on the Nintendo handhelds mentioned above. WarioWare Smooth Moves showcases the many stances and motions the Wiimote can support without the motion plus. Wii Sports Resort is a great showcase of the motion plus technology.
- iPhone.Like the Wii, the original iphone initially only featured 3 accelerometers to sense tilt. Gyroscope technology was added in later models. Switching between portrait and landscape is a popular function for non-gaming applications while the sensors are mainly used for tilting in games.
- PlayStation Sixaxis & Move.In reaction to the announcement of the Wii controller, Sony supposedly introduced motion controls into their controller. This technology was implemented into a games to varying degrees with varied success (e.g. Warhawk, Lair, Uncharted, Flower, Echochrome). In reaction to the Wii’s success Sony developed their own brand of pointer-motion control. Though the tech is a bit different than the Wii’s, the over all function is mostly the same.
- 3DS/Wii U/Vita.The next generation of Nintendo and Sony’s gaming devices all feature motion control technology. Interestingly, all three of these devices are handhelds featuring buttons, a touch screen, speakers, and an independent power supply.
The great strength of motion controls doubles as its greatest drawback. It doesn’t get much more intuitive than performing an action in a video game exactly like you would in real life (assuming you have experience with the action). Likewise, controlling a 3D action in a video game can be the most intuitive using a controller capable of sensing 3D space or forces. Instead of breaking apart or abstracting the action so that it’s controlled with a set of buttons, triggers, or sticks controlling pitch, yaw, speed, elevation, etc., a 3D controller can control the action in one, complete, and possibly buttonless way.
The downside is that tapping into the wealth of accumulated knowledge via intuitive controller design also runs a greater risk of tapping into stronger expectations. If you hand someone a Wiimote and tell them to swing it like a sword, chances are they’ll swing it like a cartoon sword. With no sense of weight, momentum, or clashing physical bodies a disconnect between player input and game output can emerge. It’s not like this issue is unique to motion controls. As I said with buttons, it’s common to hit buttons or use any other input device and get zero reaction/feedback because of what’s happening in the game. The only difference is, with motion controls user often blames the controller. The benefit of abstracting actions and mapping them to buttons and the like is that a unique bridge is created between the user and the game where the user suspends disbelief and calibrates expectation. Such is the nature of dealing with abstractions from video games to story telling. This natural calibration process may not go over so smoothly if the player is convinced that they can do the real life action well (even if they cannot). Instead of meeting the game halfway, such player make excuses and grow frustrated.
Motion controls also have to work with limitations that are similar to touch screen, mouse, and microphone controls. Like touch screen interactions, with motion controls there is no physical limit or tactile feedback for your motions. This means knowing where to start and stop is a matter of understanding the motion/mechanic you perform. This is inherently more complex. Like a mouse, there doesn’t have to be a re-centering of the motion device, though many games have certainly developed such features on the software side. Because motions are relative to the device, you may have to work at pre-positioning the device to best prepare for upcoming actions. If a game doesn’t feature any kind of motion sensitivity calibration, there’s no way to avoid the “pick up and place” issue PC gamers have avoided with high mouse sensitivity. Depending on the game, being out of position can really clash withplayer expectations. And like the microphone there is no clear or common “off” state for motion controls. This can be troublesome when players try to re-center or reposition the device and inadvertently perform another actions. This issue is very similar to buttons designed with a negative edge discussed in part 1.
Furthermore, there are many different ways the system can use the data from the controller. At any point during the input window, the system can look at any individual sensor or a combination of the controller angle in 3D space, direction of motion in 3D space, and speed. Because a motion made by the player can simply serve as a digital button or a fully analog input along 3 axis, knowledge of how the system works is a crucial factor in understanding how player expectation shapes their experience. And if the system isn’t calibrated property potential issues increase.
The good news is that motion controls can be more intuitive, engaging, and analog than other input devices. With 3 independent analog axes controllers can be tilted and moved in any direction. The sensitivity and range is great, and motion controls can sit invisibly behind other controller types and functions. For example, in New Super Mario Bros Wii players play with buttons and a D-pad like the original game. However, there are a few elements in the game that are controlled via Wiimote tilt. At these moments, there's no need to switch controller grips or reposition your hand. The motion controls simple kick in and you're already doing it. Combining traditional and motion controls together is a powerful combination. Notice how none of the devices I listed above exclusively feature motion controls. In other words, while playing with buttons, sticks, or even touch screens, motion controls can be at the ready to enhance your experience.
The following are well design and otherwise notable examples of motion controls.
- Shake to replace a button. Many have scoffed at motion controls that essentially swap a button press for a motion. I see nothing wrong with this kind of design. In some cases a button might be better, but not all cases. In the simplest cases, the motion-button is more engaging. Twilight Princess sword swing. Wario Ware Smooth Moves actions. Wario Land Shake It actions. Super Mario Galaxy 1 & 2 spin punch. Pixel Junk Eden dive attack. Mario Strikers Charged tackle. Donkey Kong Country Returns's somewhat controversial shake-to-roll design is actually better than a button press. More on this later.
- Analog tilt replacing analog sticks. By using more muscles than those in your thumb, 2D analog tilt controls are generally easier to be more precise. Monkey Ball Banana Blitz general controls. Super Mario Galaxy 1&2 mini game type levels. Excite Truck, Mario Kart Wii, and Flower for steering.
- Angle detection. These games use the sensors to detect the angle the controller is held at instead of sensing motion. Examples include Samba De Amigo poses. In Guitar Hero activate star power by tilting the guitar. To charge missiles in Metroid: Other M hold the Wiimote straight up then hold the A button.
- 3D analog motions. When I first realized that I had full 3D control over how I returned volleys in tennis, I was blown away. The height, the angel of the hit, the power behind it, and the top-side-back spin were all the result of how I swung the Wiimote. This design is far more advanced than most understand or give the game credit for. I had to buckle down and really focus on my technique to earn a platinum metal in the training mode. And this technology only got more precise with the Wii Motion Plus and Wii Sports Resort.
Camera
Similar to motion controls, almost every current generation and upcoming gaming device features a built in camera or some kind of camera peripheral. Taking pictures and video chat are common features, but few games use the camera as a control input device. For the final piece of input technology I'll cover in this series, we're looking at camera. This won’t take very long.
A camera input device is sensitive to light. By analyzing pictures or video, the software is able to recognize patterns and track movement, shapes, colors, and even depth with infrared technology. Since just about every gaming device supports a camera and they all work about the same, I'll skip right to the pros and cons.
The pros include the potential for controller free gaming. Keep in mind that, like microphone controls, cameras can tap into the vast range of body language and body motions. Also it's possible to scan in objects for the camera to track. Augmented reality is a type of interactive experience where the camera sees into the world, displays what it sees on screen, and alters the image in some way.
Unfortunately, the cons are many for camera control. Real-time image processing and augmentation is much more processor intensive than any other input device. Think of this as the difference in the information pipeline between streaming a video off the internet and flicking on a light switch. Other issues include lighting. If there's too much, too little, or specific kinds of light in the environment (e.g., sunlight) some camera technologies falter. Also, camera resolution is important. The higher the resolution of the incoming image the clearer a picture the system has to work with (at the expense of increased processor consumption).
The following are a few examples of camera control.
- Playstation Eyetoy.
- Wario Ware: Snapped. This DSiWare game tracks your head and hands to play microgames.
- Ubisoft Your Shape.
- Looksley's Line Up. This game uses face tracking technology to create the illusion of looking into a window into a game world. 2D, button free, analog controls are possible using camera technology like this.
- 3DS & Vita ARGs. With software designed to recognize specific symbols, the camera is used to modify images of reality. Continually updating the images also turns the handheld into a 3D controller. Move up, down, sideward, or closer and the visual update accordingly. Check out Ghostwire for a example of an immersive, scary experience.
- Playstation Eye.
- Microsoft Kinect. Now you and a friend can play together controller free. Move your arms, your legs, or any other part of your body in the 3D space in front of the device to play. Kinect can even recognize you by how you look. There are many games in development for kinect.
In part 6 we'll examine how ergonomics and expections are to controller design as we beging to wrap things up and put it all together.