Master thesis: hand gesture recognition system

Master thesis

During the second half of 2012 I was working on my Master thesis with the topic "Building an appearance-based hand gesture recognition system using the Microsoft Kinect". This happened at Fraunhofer FIT, where I was employed as student worker at that time. FIT had previous expertise in hand and finger-tip recognition by the award-winning 3DMT work by Georg Hackenberg, done in 2010. In his work he used the SwissRanger SR4000 TOF depth sensor, a device that is not affordable to end-users, still priced at more than 4000 USD.

My initial work was to extend 3DMT, adding support for the affordable Kinect sensor. It turned out finger-tips (and more generally, hand poses) are not easily identified at distances beyond 1.5 m between hand and sensor, due to limited depth image resolution. I then explored the use of skin-color classification from the high-resolution RGB image, all attempts showing major deficiencies in real-world environments. The goal to maximize the interaction space beyond 1.5 m with the Kinect could not be reached. I then developed a depth-data-based hand pose training and recognition system. Using a graphical tool, the user can design a set of hand poses (think: victory, fist, palm, ...), show the poses to the system (=train it) and finally get a working classification system that can be integrated into any C++ application.

The video below illustrates the procedure and also demonstrates a kiosk-application built on top of the classification. This application named "FIT Xplorer" allows guests to navigate through a catalog of employees and projects of the Fraunhofer FIT institue, using hand gestures. It was designed in a user-centered approach where the users were involved in both requirements, prototype testing and evaluation of the final prototype.

Android game: Looping Louie

Looping Louie

The Android game Looping Louie is an adaptation of the board game (take a look at this YouTube demonstration) to the Android platform. The real game is not only played by children but also by teenagers and adults who use it to consume alcohol at an increasing pace Wink.

With a team of 5 fellow students we created an Android port of Looping Louie in the winter term of 2012, as part of the SCRUM lab held at Fraunhofer FIT. Each player uses her own device, and pressing the lever is mapped to a shake-up movement of the phone, which has to be made right when Louie's plane passes by. With all devices in the same WiFi network (using existing infrastructure or the Tethering-functionality of a phone), anyone can create a new game session which others can join. Once all players are ready, they start the game together, kicking Louie with their own lever until all players but one have lost their chickens. The SCRUM process was used throughout the creation of this game.

Claim your Station

Claim your station

Claim your station is a location-based Android game I developed with a fellow student during the 2011 winter term, for the MIxed-reality games for mobile devices lab held at Fraunhofer FIT. This Android game makes use of the GPS and NFC (Near Field Communication) capabilities of modern smartphones.

The idea is to detect the NFC-enabled Touch&Travel Touch points that Deutsche Bahn installed at almost all railway stations throughout Germany. These points have an RFID chip and thus an NFC-phone can read the ID of the chip. Their geographic distribution isn't perfect: not every station has these touch points, some have just one, some have more than 30 of them. In the game you can "check in" at these touch points by bringing your phone close to them (within few centimeters).

Because you are part of a team, if you manage that your team holds the relative majority of check-ins at the station, that station is then claimed by your team, and your team will get points for every minute the station belongs to you. Boring waiting times that results from waiting for the arrival of the next (connecting) train present the perfect opportunity to play the game.

The game is not available on Google Play due to being an unfinished prototype. You can, however, take a look at a few screenshots below.

Kinect game: ShadowBall

Shadow Ball

ShadowBall is a mixed-reality Kinect game for PC that was developed in a team of 4 fellow students and I, as part of the user-oriented design lab at Fraunhofer FIT. The project took place in the 2011 summer term (April - July 2011). The focus of the lab was to continuously develop and improve prototypes based on user interviews and test sessions, where users interact with the prototypes.

To get an idea about the game, see the video below which explains it faster than words. Rather than creating a mouse&keyboard-based game, the user's whole body is part of the gameplay experience. The user stands in front of the Microsoft Kinect, which captures the body which is shown as semi-opaque silhouette (shadow) on the screen.

In a nutshell, the game is similar to flipper consoles, except that you hit the ball with your arms and legs which are solid,  acting as reflectors to the ball. The goal is to hit the ball at the right angle so that it hits a target area (i.e., falls into a hole). Some target areas are not appropriate for all balls, thus we used a jungle-theme where balls are represented by fruits or meat, the correct target areas being represented by carnivores or herbivores respectively.

LinkSmart Demo

While being a student worker at the User-Centered Computing group of Fraunhofer FIT in 2011, my colleagues and I worked on a home-automation Android app. Electrical devices are connected via smart plugs which can be remotely controlled by the user. The user is always aware of the current power consumption and which devices are turned on. A live camera feed allows the user to monitor their home. We developed the Android app, an HTML+JS Viewer (based on Google Web Toolkit) and the Java-based server software that enabled anyone to control a few devices that were installed in a demonstration room at Fraunhofer FIT. A demo video can be seen on this Fraunhofer FIT web site.

Contributions to Nexuiz

Nexuiz (Classic) is a free, open-source First Person Shooter (FPS) game, similar to commercial games such as Unreal Tournament and Quake. The scene around it was most active between the launch in 2005 and the beginning of 2011. At that time the game was simply called "Nexuiz" but then the core development team decided to create a spin-off commercial game, which is now officially referred to as Nexuiz, using the CryEngine. The greater part of the community and many developers who insisted on the open-source idea of the game decided to fork Nexuiz and to create the game Xonotic. From forum analysis, its community is shrinkin since mid 2013, nevertheless it is an entertaining, free FPS worth giving a shot!

Between 2008 and 2010 I contributed to (the original) Nexuiz as GreEn`mArine in various ways:

I was very interested in competitive 1-vs-1 and team-games. With the help of other community members, I hosted several tournaments in Europe and created a PHP-based ladder website where interested players could register and challenge others in different game modes (both websites are no longer used, but do still work). I also contributed a few features to the game, related to competitive gaming, such as a warm-up mode where people would ready up to begin the game, and the ability to call time-outs.

In 2009 I hosted a video podcast called Nexuiz In-Depth (click here to get to the YouTube channel). In about 10 episodes I explained aspects of the in-game movement as well as the usage of many hidden commands that I discovered while doing programming work on the core.

Yet another contribution was the Nexuiz Demo Recorder. This is a Java tool (with a graphical user interface) that can be used to automate the job of mass-recording videos from Nexuiz or Xonotic. Both games allow to record your play sessions in a compressed format called "demo", which doesn't actually contain video, but the sequence of events that happened in the game (e.g. when you fired a rocket at second 33).

The game engine provides the means to replay these demos at arbitrary speed and to dump them to video files (e.g. uncompressed avi). Using that feature is cumbersome since you have to manually start the recording process, i.e. press a button once you want your video to start, and likewise when you want it to end. The Nexuiz Demo Recorder allows to specify (many) recording jobs in advance, where you specify the file name of the recording, start second, end second and destination video file name. Then the program will launch the game and batch-process all your jobs. Thus, it is a huge time-saver for anyone who creates machinima movies, such as “fragmovies” that demonstrate the skill of individual players or teams.

Nexuiz & Tactical Ops Movies

Before starting to play Nexuiz in 2008 I had played the game Tactical Ops: Assault on Terror for 5 years. Tactical Ops can be described as an arcade-version of the classic Counter-Strike game. At the time the game was very popular and the "clan wars" (=matches between two clans/teams) took places in leagues and tournaments, including ClanBase, ESL (Electronic Sports League) and GameStar Liga.

Tactical Ops also allowed to record games in the demo format and to export its replays as videos. This kindled my interest and so I created a total of 3 "frag videos" featuring players from my clans (including me) battling (= fragging / taking down) other teams.

Having created the Nexuiz Demo Recorder (see above) I also used it to create two Nexuiz videos (which were never published).