3D User Interfaces


Scientific Article- Writing for ee4fn, cs4fn or Audio!<br /><br />
ee4fn, cs4fn and Audio! are QMUL magazines produced in hard copy and online for secondary school students across the UK. Their aim is to teach students about topics in the field of Electronic Engineering and Computer Science and to inspire them to find out more and go on to further s tudy in the subject. Your task is to write a short piece for one of these magazines.<br /><br />
What will you write about?<br /><br />
In week 2 you will be offered a number of Topic Tasters, and then you will choose one of these Topics to research. Now for your article, you should choose an idea or concept or issue that is part of your research Topic. Your job is then to communicate this to the magazine readers in ways that will interest and enthuse them.<br /><br />
Professor Paul Curzon is the editor of the ee4fn, cs4fn and Audio! mag azines and has written extensively for them himself. You are encouraged to listen to him talking about how to go about writing an article. He has produced 3 short resources that are on the ECS4020U QMPlus site.<br /><br />
Amongst other things he suggests that a successful article often has the following features.<br /><br />
1) One main point or message, and at most 3 sub-points<br /><br />
2) A title that is likely to attract the interest of the readers<br /><br />
3) A ‘grab’ – something like a question or a surprising statement in the first couple of sentences<br /><br />
that will make the reader want to read on.<br /><br />
4) A ‘pull-together’ ending (probably linking back to the ‘grab’)<br /><br />
5) Avoidance, where possible, of jargon. Where jargon can’t be avoided, it is explained for the<br /><br />
reader.<br /><br />
6) Metaphors or links to everyday things that people know about<br /><br />
7) A story-like structure that links the piece together<br /><br />
Bibliography<br /><br />
Your article should also be accompanied by a correctly presented bi bliography in the Harvard style. Information and materials about writing bibliographies and referencing for university assignments will be given in the relevant lectures and will be available on QM+.<br /><br />
Formatting and presentation<br /><br />
Please comply with the instructions below<br /><br />
? ThecompletedarticleshouldfitintoonesideofA4(ie.apageofa CS4fnmagazine)<br /><br />
? Font Arial 11 point<br /><br />
? Single spaced<br /><br />
? It should include your name and student number as well as the title of the article<br /><br />
? The bibliography should appear on a separate page to the article itself<br /><br />
ECS402U Professional and Research Themes<br /><br />
Marking<br /><br />
70% and above<br /><br />
An excellent article that might be considered for publication in cs4Fn, ee4fn or Audio!, with only fairly minor editorial changes. The writer appears to have followed closely the advice given by the magazines’ editor, Paul Curzon and conveys understanding and enthusiasm for an interesting or useful concept, idea or issue in the field of Computer Science or Electronic Engineering. The article presents the idea/issue is a way that is engaging to the intended reader. It uses techniques to grab and hold the reader’s attention and may make lively and appropriate us e of metaphor or everyday examples to help communicate. There is a sustained focus without rambling or repetition. The writing flows well, is grammatically fluent and is almost entirely free of errors in spelling and punctuation. Acorrectlypresentedbibliographyisincludedthatshowsevidencethatthereaderhas accessed a number of different sources in the writing of the article.<br />
<br />
<br />
<br />
The scientific article should be written about the 3D User Interfaces. The file is attached (about 3D interfaces)

a) b) c) d)
Figure 1. SpaceTop affords a) 3D direct spatial interaction, b) 2D direct touch, c) 2D indirect interaction, and d) typing. It aims to
meld seams between these modalities by accommodating them in the same unified space and enabling fast switching between them.
ABSTRACT input through a mouse or a touchscreen. While GUIs have
SpaceTop is a concept that fuses 2D and spatial 3D many advantages, they can constrain the user due to the
interactions in a single desktop workspace. It extends the limited screen space and interaction bandwidth, and there
traditional desktop interface with interaction technology exists situations where users can benefit from more
and visualization techniques that enable seamless expressive spatial interactions. For instance, switching
transitions between 2D and 3D manipulations. SpaceTop between overlapping windows on a 2D screen adds more
allows users to type, click, draw in 2D, and directly cognitive load than arranging a stack of physical papers in
manipulate interface elements that float in the 3D space 3D space [7]. While there has been advances in sensing
above the keyboard. It makes it possible to easily switch and display technologies, 3D spatial interfaces have not
from one modality to another, or to simultaneously use two been widely employed in everyday computing. Despite
modalities with different hands. We introduce hardware and advantages from spatial memory and increased
software configurations for co-locating these various expressiveness, potential issues related to precision and
interaction modalities in a unified workspace using depth fatigue make 3D desktop computing challenging.
cameras and a transparent display. We describe new We present SpaceTop, an experimental prototype that
interaction and visualization techniques that allow users to brings 3D spatial interaction space to desktop computing
interact with 2D elements floating in 3D space and present environments. We address the previously mentioned
the results from a preliminary user study that indicates the challenges in three interdependent ways. First, SpaceTop
benefit of such hybrid workspaces. accommodates both conventional and 3D spatial
Author Keywords: interactions in the same space. Second, we enable users to
3D UI; Augmented Reality; Desktop Management switch between 3D I/O and conventional 2D input, or even
use them simultaneously with both hands. Finally, we
ACM Classification Keywords present new interaction and visualization techniques to
H.5.m. Information interfaces and presentation (e.g., HCI) allow users to interact with 2D elements floating in 3D
General Terms space. These techniques aim to address issues and
Human Factors; Design; Measurement. confusion that arise from shifting between interactions of
different styles and dimensions.
Desktop computing today is primarily composed of 2D RELATED WORK
graphical user interfaces (GUI) based on a 2D screen with Previous work has explored 2.5D and 3D representations to
better support spatial memory in desktop environments [1,
7]. Augmented Reality systems exploit the cognitive
Permission to make digital or hard copies of all or part of this work for benefits of co-locating 3D visualizations with direct input in
personal or classroom use is granted without fee provided that copies are a real environment, using optical combiners [8, 6, 5]. This
not made or distributed for profit or commercial advantage and that copies makes it possible to enable unencumbered 3D input to
bear this notice and the full citation on the first page. To copy otherwise, directly interact with situated 3D graphics in mid-air [5, 9].
or republish, to post on servers or to redistribute to lists, requires prior
specific permission and/or a fee. SpaceTop extends these concepts with an emphasis on
CHI 2013, April 27–May 2, 2013, Paris, France. streamlining the switching between input modalities in a
Copyright © 2013 ACM 978-1-4503-1899-0/13/04…$15.00.


Session: 3D User Interfaces CHI 2013: Changing Perspectives, Paris, France
unified I/O space, and the combination of such 3D spatial
interaction with other conventional input modalities, to
enable new interaction techniques.
Other related research explores the transitions between 2D
and 3D I/O by combining multi-touch with 3D direct
interaction [3], or through 2D manipulation of 3D
stereoscopic images [4], with an emphasis on collaborative
interaction with 3D data, such as CAD models. Our work
focuses on how daily tasks, such as document editing or
task management, can be better designed with 3D spatial
interactions in existing desktop environments.
Figure 2. SpaceTop hardware configuration (left).
SPACETOP IMPLEMENTATION Hand tracking and pinch detection (right).
At first glance, SpaceTop looks similar to a conventional
desktop computer, except the transparent screen with
keyboard/mouse behind it. Users place their hands behind The setup also detects if the user’s hands are touching the
the screen to scroll on the bottom surface, or type on the 2D input plane based on the technique described in [10].
keyboard. Through the transparent screen, users can view INTERACTION AND VISUALIZATION
graphical interface elements appearing to float, not only on 2D in 3D: Stack Interaction
the screen plane, but also in the 3D space behind it or on the In SpaceTop, graphical UI elements are displayed on the
bottom surface. The users can lift their hands off the bottom screen or in the 3D space behind it. In our scenarios, details
surface to grab and move floating windows or virtual or 2D views of 3D elements are shown on the foreground
objects using “pinch” gestures. plane (coinciding with the physical screen). While objects
We accommodate 3D direct interaction, 2D touch and can take various forms in 3D space, we chose to focus on
typing, with an optically transparent LCD screen and two window interaction and 2D content placed in 3D space,
depth cameras (Figure 2) in a 50×25×25 cm3 volume. such that the system can be used for existing desktop tasks.
Another advantage of the window form factor in 3D is that
Display: Prototype LCD with per-pixel transparency
it saves space when documents are stacked. It can, however,
We use a display prototype by Samsung, designed to show
become challenging to select a particular window from the
graphics without backlights in contact with its transparent
dense stack.
LCD. The 22″ transparent LCD displays 1680×1050 pixels
We designed various behaviors of stacks and windows to
images at 60 Hz with 20% light transmission. It provides
ease retrieval in stacks, as illustrated in Figures 3a–f. Users
maximum transparency for white pixels, and full
can drag-and-drop a window from one stack to another, to
opaqueness for black pixels. We use the unique per-pixel
cluster it. As the user hovers his finger inside a stack, the
transparency to control the opacity of graphical elements,
layer closest to the user’s finger gets enlarged and more
allowing us to design UIs that do not suffer from the
opaque. When the user pinches on the stack twice, the
limitations of half-silver mirror setups, where pixels are
dense stack expands to facilitate selection. The surface area
always partially transparent. We ensure that all graphical
below the stack is used for 2D gestures, such as scrolling.
elements include clearly visible opaque parts, and use
Users can, for example, scroll on the bottom surface of the
additional lights for the physical space behind the screen, to
stack to change the order of the documents in the stack.
improve the visibility of the user’s hands and keyboard.
We designed a Grid and Cursor system to simplify the
Head and hand tracking with depth cameras organization of items in 3D. It provides windows and stacks
One depth camera (Microsoft Kinect) faces the user and with passive reference cues, which help guide the user’s
tracks the head to enable motion parallax. This allows the hands. The cursor is represented as two orthogonal lines
user to view graphics correctly registered on top of the 3D parallel to the ground plane that intersect at the user’s finger
interaction space wherein the hands are placed. Another tips. These lines penetrate the grid box that represents the
depth camera points down towards the interaction space and interaction volume, illustrated in Figure 3a.
detects position and pinch-gestures of the user’s hands [11].

Figure 3. a) 3D grid and cursor. b), c) Sliding door. d) Highlighting. e) Scroll to change
the order of windows. f) Expansion of a stack. g) Shadow Touchpad. h) Inter-shadow translation.

Session: 3D User Interfaces CHI 2013: Changing Perspectives, Paris, France

a) b) c)
Figure 4. Users can spatially manage windows in 3D space,
and leverage spatial memory to retrieve tasks. Figure 5. a) Selecting a piece of text and dragging it out of
the source document. b) Text appears to be floating in open
Modeless Interaction space while being moved from one document to another.
Our guiding principle for designing high-level interfaces c) The text appears again in the destination document.
and visualizations is to create a seamless and modeless
workflow. Experiments have shown that when users shift behind the main task, allowing users to rely on their spatial
from one interaction mode to another, they have to be memory to retrieve them. This spatial persistence mitigates
visually guided with care, such that the user can mentally some of the cognitive load associated with conventional
accommodate the new interaction model. Particularly, task management systems. Sliding door or stack interaction
smooth transition between 2D and 3D views, and between can be directly applied to categorize, remember, and
indirect and direct interactions are challenging, since each retrieve tasks (Figure 4).
of them is built on largely different mental models of I/O. Bimanual, Multi-fidelity Interaction
Sliding Door: Entering the Virtual 3D space Interesting interactions arise when each hand is interacting
In the 2D interaction mode, the user can type or use a in different styles and fidelities. The following applications
mouse or touchpad to interact with SpaceTop, as in any demonstrate the potential of such bimanual, multi-fidelity
conventional 2D system. When the user lifts her hands, the interaction.
foreground window slides up or fades out to reveal the 3D Document Editing Scenario
space behind the main window. When the hands touch the When composing a document, the user often needs to copy
bottom surface again, the foreground window slides down portions from other documents, such as, previous drafts or
again, allowing users to return to 2D-mapped input. The outside sources. SpaceTop allows the user to use the
sliding door metaphor can help users smoothly shift focus dominant hand to scroll through a main document, while
from the “main” 2D document to “background” contents simultaneously using the other hand to quickly flip through
floating behind (See Figures 3b-c). a pile of other documents, visualized in 3D space, to find a
Shadow Touchpad: One Touchpad per window relevant piece of text. The user can then drag that piece of
Touchpad interaction with 2D windows floating in 3D text into the main document through the more precise
space introduces interesting challenges. Especially when touchpad interaction. In this way, SpaceTop allows users to
working with more than one window, it is not quickly switch back-and-forth between low-bandwidth ,
straightforward how to move a cursor from one window to high-precision interactions (copying lines) and high-
another. Indirect mapping between the touchpad and the bandwidth, low-precision interactions (rifling through
window can conflict with the direct mapping that each documents), or use them simultaneously.
window is forming with the 3D space. To address this issue, 3D Modeling Scenario
we propose a novel concept called Shadow touchpad, which While 3D spatial interactions provide means for the user to
emulates a 2D touchpad below each of the tilted 2D materialize their design through spatial expression, much of
documents floating in 3D space. When a window is pulled the interaction in CAD require precise manipulation and is
up, a shadow is projected onto the bottom surface, whose controlled in 2D. SpaceTop allows for natural transitions
area functions as a touchpad that allows the user to interact between these interaction modes. The user can start
with that window. When multiple screens are displayed, prototyping a model with free-form manipulation. Once
each of them has its own shadow touchpad area. fine control is required, the user can select a surface of the
Inter-shadow translation of 2D element 3D model and pull up an editing console to the foreground

Users can move 2D objects (e.g., text and icons) from one
window to another by dragging the object between the
corresponding shadow areas. The object will be visualized
as a floating 3D object during the transition between the
two shadow touchpads, similarly to the balloon selection
technique [28], as shown in Figure 5.
Task Management Scenario
Effective management of multiple tasks has been a central
Figure 6. Modeling Applications. The user can pull up a
challenge in everyday desktop computing. In SpaceTop, the
face to the foreground windows, edit it on the 2D
background tasks occupy a fixed position in the 3D space screen, and manipulate in 3D space.

Session: 3D User Interfaces CHI 2013: Changing Perspectives, Paris, France
of the screen. The user can then precisely modify some insights for future improvements. A few users also
dimensions by dragging a side, typing a number, or choose commented that they might perform better with a
material properties by touching a 2D palette on the ground. stereoscopic display, in addition to the aid of the grid and
cursor. Although previous work indicates that stereoscopy
Ten participants (age 19–29, 2 female) were recruited from has limited benefit over monoscopic display with motion
a university mailing list, none of whom had previous parallax [5], we plan to also explore a stereoscopic version
of SpaceTop. We think that the visual representation could
experience with 3D user interfaces. They were able to
be better designed to provide users with clearer guidance.
familiarize themselves with the system until they performed
While the current configuration allows us to rapidly
each action comfortably (3–6 min). The total experiment
time for participants was between 70-80 min. prototype and explore interactions, we plan to improve
ergonomics and general usability with careful design of the
Switching between indirect 2D vs. direct 3D interaction physical setup.
12 partially overlapping colored windows (red, green, blue,
or yellow), containing a shape (triangle, square or star), CONCLUSIONS AND FUTURE WORK
were shown. Participants were given tasks, such as “grab SpaceTop is a concept that accommodates 3D and
the yellow square and point to its corners”, or “trace the conventional 2D (indirect/direct) interactions in a single
outline of the blue triangle”. They performed four different, workspace. We designed interaction and visualization
randomized tasks for three spatial window configurations, techniques for melding the seams between different
for a total of 12 trials for each of two blocks. The SpaceTop interaction modalities and integrating them into modeless
block used spatial window placement with head-tracking workflows. Our application scenarios showcase the power
and participants used a combination of gesture, mouse and of such integrated workflows with fast switching between
interactions of multiple fidelities and bimanual interactions.
keyboard interaction, for constant switching between
We believe that SpaceTop is the beginning of an
typing, 2D selection and 3D interaction. In the baseline
block, windows were shown in the display’s 2D plane and exploration of a larger field of spatial desktop computing
only mouse and keyboard interaction was available. interactions and that our design principles can be applied to
a variety of current and future technologies. We hope that
Questionnaire responses (5-point Likert scale) indicate that this exploration offers guidelines for future interaction
the SpaceTop interactions were easy-to-learn (3.9). designers, allowing better insight into the evolution of the
Participants did however find it slower (3.2 vs 4.2) and less everyday desktop experience.
accurate (3.2 vs 4.6) than the baseline. Users’ comments
include “after I repeated this task three times (with the same
We would like to thank Akimitsu Hogge and John Weiss
arrangement), my arm starts moving towards the target
for assistance in implementation, and Steven Bathiche,
even before I see it”, “switching to another window is as
Vivek Pradeev, Otmar Hilliges, Andy Wilson, Hrvoje
simple as grabbing another book on my (physical) desk”.
Benko, Pranav Mistry, and Microsoft Applied Sciences for
Another user commented that the physical setup constrains
valuable discussions.
his arm’s movement which makes him exhausted easier.
Text editing: Search and copy/paste 1. Agarawala, A., and Balakrishnan, R., 2006. Keepin’ it real: pushing the
Participants skimmed the contents of six different document desktop metaphor with physics, piles and the pen. CHI ’06, 1283–1292.
pages placed in the 3D environment. They were then asked 2. Benko, H., and Feiner, S. 2006. Balloon selection: A multi-finger technique
for accurate low-fatigue 3d selections. 3D UI ’06, 79–86.
to find a specific word and pick-and-drop it into the 3. Benko, H., Ishak, E., and Feiner, S. 2005. Cross-dimensional gestural
document on the foreground screen (see Figure 5). interaction techniques for hybrid immersive environments. VR ‘ 05, 209–
Six participants commented that it felt compelling to be 4. Hachet, M., Bossavit, B., Cohé, A., and Rivière, J., 2011. Toucheo:
able to quickly rifle through a pile of documents with one multitouch and stereo combined in a seamless workspace. UIST ’11, 587–
hand while another hand is interacting with the main active 5. Hilliges, O., Kim, D., Izadi, S., Weiss, M., and Wilson, A. 2012.
task. One user commented: “it feels like I have a desktop HoloDesk: direct 3d interactions with a situated see-through display. CHI
computer and a physical book next to it”. “This feels like a ’12, 1283–1292.
6. Olwal, A., Lindfors, C., Gustafsson, J., Kjellberg, T., and Mattsson, L.
natural role division of right/left hand in the physical 2005. ASTOR: An autostereoscopic optical see-through augmented reality
world”. Three users reported that they had a hard time system. ISMAR ’05, 24–27.
switching their mental models from 2D indirect mapping 7. Robertson, G., Czerwinski, M., Larson, K., Robbins, D., Thiel, D., and
Dantzich, M. 1998. Data mountain: using spatial memory for document
(touchpad) to 3D direct mapping (spatial interaction), which management. UIST ’98, 153–162.
occurs when the user tries to drag a word out of a shadow. 8. Schmandt, C. 1983. Spatial input/display correspondence in a stereoscopic
computer graphic work station. SIGGRAPH ’83, 253–261.
DISCUSSION 9. Treskunov, A., Kim, S. W., Marti, S. 2011. Range Camera for Simple
Users’ comments suggest that fast switching and bi-manual behind Display Interaction. IAPR MVA ’11, 160–163.
10. Wilson, A. Using a depth camera as a touch sensor. ITS ’10, 69–72.
interaction provide compelling experiences, and that they 11. Wilson, A. 2006. Robust computer vision-based detection of pinching for
can benefit from spatial memory (task 1). We also gained one and two-handed gesture input. UIST ’06, 255–258.



3D User Interfaces