Controller Design: Pointers pt.4

part 1. part 2. part 3. 

Personal computers wouldn't be the same without the mouse. Instead of navigating via key strokes, with a mouse users can simply point to the on screen interface to indicate which actions to take. It's hard to imagine using a PC without a mouse, especially for PC games. The mouse's unique features put it into a class of few. It along with the keyboard is the main reason why there is such a large disparity between the types of games that are made for PCs versus consoles. But the mouse isn't the only pointing device common to gaming.

  .

That's the Point

A pointing device either directly or indirectly uses light or other mechanisms to relay positional coordinates to a cursor via 2D or 3D movements. The cursor movement is always proportional to real life movements. Also, there is no springy action or snap back with pointers. The modern mouse is a 2D pointing device with high sensitivity. Ignoring its buttons and other features, the mouse is controlled by multiple fingers, the wrist, and the rest of the arm. With so many muscles and joints involved mouse control is great at tracing straight lines in various directions, tracing circles, making very slight movements, and stopping dead still on a target. The larger the movement the more ease one has with mouse control due to the increased number of muscles involved. Conversely, very small complex movements are not as easy to be precise. In other words, pointers are great at pointing and moving, but not great at fine control like with handwriting or stylus control. Test this out for yourself by playing Dexterity, a skill testing game by B.E.S. 

The mouse is also a relative pointing device. As long as you have a large enough flat surface (and a long enough cord/connection) you can pick up and reposition the mouse anywhere. It doesn't matter where you place the mouse, all of the pointing is relative to the changes in mouse position. To avoid having to pick up and replace the mouse for larger movements many gamers have adjusted to a high cursor speeds so that moving less in real life moves more on the screen. At the same time these limitations increase the focus players need to keep on the cursor like working with some touch screens. 

Pointers don't stop with 2D. Some devices can point in 3D. But to cover these less common devices, I might as well describe them in the following examples. 

• Wiimote. This device is the most mainstream 3D pointing device ever created. With the sensor bar set either above or below the TV, pointing is more direct than with a mouse. In other words, it is important where you hold and point the Wiimote in real space. With this design you can use some muscle memory to increase accuracy and precision. While pointing at the TV the system can not only tell your 2D screen position and 3D depth (closeness to the TV), but it can also tell the angle you point at the TV. Though the pointing is accurate, holding the Wiimote in place tends to cause the aim or cursor to tremble. This issue can be smoothed on the software side. Also, up to 4 Wiimotes can be synced to the same Wii. 
• Playstation Move. Using the Playstation Vision Cam and the Move controllers the system works as a 3D pointer much like the Wiimote technology. 
• Novint Falcon (see video). This device is a futuristic piece of hardware. First of all, it's a 3D pointing device. Secondly, using haptic feedback the system can simulate different textures, object densities, and directional forces. For example, if you get shot in an FPS you'll actually feel the controller jerk in the appropriate direction. 
• DS Side Controller (see video). Never released outside of Japan, this controller is basically a light mouse for the DS. 
• Kinect. With vision of the space in front of it, the Kinect camera can sense the players hand or any other object and track it as a pointer. The trouble with controller free gaming is that you lose access to some of the most intuitive and effective input devices like buttons and analog sticks. 

The precision and accuracy of the mouse is great, but the simplicity of control is the key. No matter how you move the mouse, the cursor moves at the same rate that you move it. In other words, you never have to calibrate for the increasing speeds that analog stick movement tends to have. With the mouse and increasingly higher resolution computer monitors many, if not most, PC game developers design their UI (user interface) with tiny text and small virtual buttons. There much more to discuss about PC game design and feel, but this is not the place. 

The mouse pointer can do little without mouse buttons. For the most part the Kinect has to work around limitation of controller free interaction. Dontclick.it web developers chose to design a website that can be navigated completely without clicking the mouse. This site is stunning, stylish, informative, and entertaining topping the chart of my favorite websites of all time. Experience it for yourself to understand just how the UI and virtual buttons had to be redesigned to compensate. 

Input devices become extensions of our hands and our minds. According to MIT research we think with our hands, eyes, arms, and anything else we can manipulate. So while we're holding a controller, it's possible to think through it. When we hold a mouse, we often unconsciously move it to where our minds are focused. Moving the mouse, seeing the cursor, and moving the mouse again creates cognitive positive feedback loops. I'll cover this in greater detail soon on this blog.  

Microphone

Voice control. This idea is nothing new. Scifi shows have featured characters conversing with their computers for a long time. I remember when voice recognition technology was spotty at best. This coupled with poor quality microphones didn't make a winning combination. Now, I can can speak just about anything into my phone like "search: jigglypuff serebii.net" and the system recognizes my speech perfectly. To reiterate, voice recognition technology finally works. As far as gaming goes, this technology is slowly gaining steam. First, I'll address the pros and cons of microphone input. 

The microphone is a device that converts the sound waves (mechanical waves) into a digital signal. This means all the different ways to alter sound become distinct input variables. This includes frequency, tone, and volume. Even before speech/language recognition comes into play, the mic can function like a hands free button or trigger. Just program a mechanic to react to the loudness of incoming sound and the game can react in an 1D digital or analog fashion. On the complex side, mics can use speech recognition to process language. The range of sensitivity, specificity, and versatility of language is unmatched by any other input method. For a comparison, speech recognition is like the complex move command in Street Fighter times a billion. The best part of all is that we already understand these billions of commands because we've internatized and continually practice using our native languages. 

On the downside, mics can pick up ambient noise that you may not be able to control. Noise filtering software and other technologies exists to help focus mics on the sounds intended to be picked up. While voice commands can replace buttons for simpler, non-timed commands, and replace analog triggers for simpler action, mics are generally not suited for these functions. Voice commands work the best with increasingly large numbers of specifics. What I mean by this is that thinking and saying words is the fastest, most intuitive method of identification when there are lots of items to choose from. Think of how simple it is to type in a question into google search. Any term. Any word. Any category. And you get results quickly. Voice commands allow for this kind of precision searching and indentification completley hands free. Just think if you could search your Pokedex for any Pokemon, item, ability, or attack just by speaking. This would be very convenient. Instead we have to push buttons, tap the screen, scroll up and down, and then click again to get to the content we want.

The following are examples that use a microphone. Be sure to check out this article I wrote on DS microphone design for more examples. 

• Donkey Kong Jungle Beat. One of last generations best platformers is controlled via a plastic bongo set. Built into the side of this unusual controller is a small microphone designed like a button to respond to loud sounds. Clapping your hands together is just the type of sound the system was designed to respond to. When you clap, Doneky Kong claps. There are even moments in the game where it's important not to clap. At these times, any loud noise in the room can set the game off.  
• Exclamation Warriors (see video). This quirky game never made it state side. Use voice commands to punch, transform, and other co-op combinations. 
• GNILLEY. Scream to survive in this indie game. Noice how the screaming acts as a analog trigger. See video here. Download it here.  
• Tom Clancy's End War. This console RTS features optional voice command controls (see video). With a list of about 50 voice commands players can control all of the action. Couple this system with some controller input and the system becomes very accurate and versatile. It also helps that the voice commands have clear visual feedback (see video). 
• Brain Age 1 & 2. These games feature voice recognition with colors, Rock Paper Scissor hands, and the secret phrase "glasses glasses." (see video)
• Kinect. The DS has a built in mic and uses it often enough. The Wii U controller has a mic as well and will support video chat. But it's the Kinect that's currently pushing voice commands in gaming the hardest. Menu navigation, weapon custimization, and ally commands are all coming. 

In part 5 I'll cover motion controls and camera based input systems. 

Comments

[Mathanas]

This is AMAZING!. What an ingenious teaching that also can benefit any learner. It explains well in a way that no one will have any question in regard to computer devices. Thank you so much for sharing this with us.