1. Please write a brief introduction and explanation of the activities in your lab.
The Mixed Reality Lab (MXR) at the National University of Singapore is a world centre of excellence for interactive media, and entertainment technology. The MXR Lab focuses its efforts on interactive and digital media technologies, combining the efforts of engineers, scientists, artists and designers to create the future of digital interaction. The most distinguishing characteristic of the MXR Lab is that it doesn't follow mainstream research but it focuses on challenging the core of mainstream research, finding new solutions to old problems.
The key objectives of the Mixed Reality Lab are:
- To provide multi-disciplinary creative learning environment, where engineering, design and art coexist.
- To radically change the paradigm of creative media technology transforming digital media interaction.
- To transform society by the production of commercially creative new media solutions for education, entertainment, and communication.
The MXR Lab works in the areas of education, entertainment, communication and military applications.
Education – The use of digital media in the field of education has in most cases has replicated the methods and materials present in classical education. The MXR Lab has changed the approach providing students with direct manipulation of the educational content in a way in which they can learn by manipulation and experimentation. The MXR technology allows the visualization of results in real time, providing the student the opportunity to experiment with entities and phenomena empowering learning in a more creative and engaging way. Entertainment - New media entertainment is being transformed by the availability of new technologies. In this field, the MXR Lab is taking a different approach from mainstream media producing solutions that bring families together. Through these new applications, children and the elderly can share their cultures, and learn about each other while sharing more time together. MXR also combines entertainment with learning experiences producing engaging approaches to cultural heritage communication.
Communication - While most research in communication technologies focuses on improving the quality of video and audio communication, the MXR Lab focuses on engaging other senses (touch, taste and smell) in the communication. The MXR Lab is using radical approaches to communication changing the medium itself, using materials such as food or air pressure to communicate taste of touch. Through this approach, the MXR Lab will make communication more meaningful. Military - The lab has carried out large scale R&D for Augmented Reality (AR) applications for Ministry of Defense, Singapore. The main goal of these researches is to combine ubiquitous computing technologies, new sensor technologies, and AR technologies to build prototype systems that are suitable for both indoor and outdoor environments. Potential uses include training, mission planning, rehearsal, and operations in battlefield.
2. Please give a description of the lab, such as, but not limited to, the different departments and their purpose, as well as the demographics and specialties of the members of your team.
The MXR Lab is composed by more than 40 individuals with a wide range of nationalities, disciplines, and ages. People working in the MXR Lab come from many different countries including China, India, Sri Lanka, Malaysia, Taiwan, Korea, Canada, Spain, Iran, Vietnam, Mexico and Greece. The MXR Lab draws experience from a variety of disciplines that include scientists, engineers, artists and designers. People in the MXR Lab come from the fields of electrical engineering, computer science, Sculpture, Industrial Design, Textiles, Architecture, Physics, Biology and Medicine. Finally having people in an age range that goes from 18 year old undergraduates to 50 year old senior Professors, the MXR Lab combines fresh ideas with years of experience.
3. What is your ongoing project?
Below is a description of some of the research activities currently being undertaken in the laboratory (as of summer 2010).
(a) Smell and taste communication In this research, we are developing a communication mechanism, which allows people to use the Internet to digitally send a taste or smell to a remote person. By introducing this technique, we expect people to share their preferred smell and taste experiences with their family and friends. The main motivation behind this work is to improve everyday life experiences through smell and tastes, since those senses are strongly associated with memories and emotions. This will open up entirely new experiences and communications such as tasting food from a remote location and sending an emotional taste message to a remote person. Current technologies have only explored taste and smell senses to some extent with primary scents or chemical compounds. The type of smells and tastes generated through these methods are limited and not significant for detailed communication. The fundamental knowledge for digitizing taste and smell senses are not well discovered. Therefore, we started this research from the fundamental level to gain the digital control over these two senses and to enable communication.
To achieve this, we will develop a wearable device, which can stimulate smell and taste senses to communicate remotely. This research has three modules in developments. First module is based on electrical and thermal methodologies to apply virtual taste to the tongue. Researches have found that apply small current as well as heating and cooling tip of the tongue can generate tastes. This module will develop by combining those two methods to generate tastes. Second phase of the research will investigate the possibilities of both smell and taste stimulation and communications through noninvasive brain stimulation methods. Initially using normal foods and smells the associated nucleus in the brain will be identified through Electroencephalography (EEG), Near-infrared imaging (NIR), and functional Magnetic Resonance Imaging (fMRI) imaging techniques. Chemical based day-to-day foods will be used in this step such as sugar, chocolate, wine, mint, lavender, and the like. Then ‘Transcranial magnetic stimulation’ method is proposed for the brain stimulation and it operates by inducing weak electric signals by rapidly changing magnetic fields produced by the circuitry outside. A thorough analysis and study on associated nucleus in the brain for a given smell and taste followed by stimulating those identified nucleus through magnetic stimulation to generate flavors and smells is the
method for this phase. Apart from these modules a communication module will be developed to communicate taste and smell information into the remote place.
(b) Food Media Project
We are creating a new digital food and cooking media for virtual and remote communication. We expect it to radically enhance family bonding and intergenerational communication. New communication technologies have made it easier to stay in contact with people, but as young people’s lifestyles become increasingly busier and the elderly population continues to rise, the problems of loneliness and weakening family ties are growing concerns. Our food media system builds remote family communication networks and allows for novel intergenerational interactions between family members through food and food activities.
It currently consists of three platforms:
(1) 3D food tele-generating printer that transforms eating into an interactive communication experience by creating edible layered messages. It produces food by printing layers of flavored agar pixels that are liquefied by a silicon heater and then solidify upon contact with the environment. The system prints the layered information according to drawings, photos, or texts submitted by the sender.
(2) Rice messaging printer that links today’s social networking platforms with traditional physical communication favored by the elderly. Our system allows young people to send a digital message via a mobile phone or website which the elderly can then receive as a familiar and comforting printed rice message. Rice is one of the most widespread foods in the world, and in Asia especially, has a deep and longstanding cultural history shared by both the elderly and young.
(3) Co-cooking devices, a sensor-embedded pot and spatula that encourage collaboration by letting elderly family members to remotely teach and co-cook with younger members. The system not only helps to replicate similar tasting dishes across remote locations, but creates the sense of physical co-presence and bonding for older generations through haptic feedback. We intend to encourage people to communicate and interact closely and emotionally through food media. The proposed food media is not only expected to facilitate peoples’ communications for staying relationships closely and the sharing of their taste experiences, but also to enhance and sustain communication between remotely-located peoples with more emotional and soft communication ways.
(c) Novel approach to Large Scale Augmented Reality
One of the key problems in Augmented Reality (AR) is to track the viewpoint of the user so that the augmentations can be properly inserted. In the past this has been done by physically modifying the environment using special targets. However, recently progress has been made in using natural features to perform AR tracking, which does not require any modification of the environment. This requires two steps, first a process (which need not be real-time) to build a map of features for the environment to be modeled. Then the second step uses these maps (which are called environment maps) along with a real-time augmentation program to draw the augmented content at the proper locations in the input video.
Currently, the most successful AR tracking systems are based on a combination of structure from motion (SFM) and the simultaneous localization & mapping (SLAM) approaches that both model and localize at the same time. Our method, by contrast, has a distinctive off-line modeling step based on the Bundler system (Structure from Motion for Unordered Image Collections). The idea is to apply this successful model building technology of Bundler to create various 3D maps of the environment, called environment maps. Then these previously constructed 3D maps will be used for AR localization over a wide area. This makes it possible to create a model using any digital camera, and then use a completely different video camera for the on-line augmentation.
The online augmentation program will use modern feature matching technology based on SIFT (Scale-invariant feature transform) / SURF (Speeded Up Robust Features) features, but these will be computed in real-time with hardware assistant from a Graphics Processing Unit (GPU).
The final online AR tracking system will be very robust over different lighting conditions, and a wide range of scales. When combined with an inertial sensor, the tracking will also be robust to rapid head motions, and will handle a large number of environment maps. The resulting system will be much more flexible and powerful than current AR tracking systems.
Key features, novelties, and steps of this method:
• Off-Line Large Scale Augmented Reality
• No Dependency on Camera
• Capable of Tracking both Outdoors and Indoors
• Robust Feature Descriptor
• Better Pose Refinement due to combination of Vision Information and Inertial Sensors input.
(d) Self-cutting paper interface
As the rapid development of digital technology is turning our society into paperless gradually, research has also shown that there are still rich advantages of paper, this traditional craft material, in our daily life. Paper is a creative medium, which is difficult to replace with standard computer interfaces: easy to use, lightweight, in expensive, ubiquitous. There is a social affordance of traditional paper craft objects, such as being bestowed as gifts, kept as souvenirs, cherished and appreciated for months or years, which is hard to attain in realm of pure software.
Paper-cutting, as one of the popular paper craft and the main method to generate shapes of paper, has been widely used by people from different cultures. Known as the inventor of paper, Chinese people have been using paper-cutting as a way of illustrating their folk tales and symbols to express good wishes, happiness and love throughout the year. Today manual activities based on paper-cutting, such as paper pop-up, haven been included in the curriculum.
Pop-ups, when done cleverly, add not only to illustrations, but to enjoyment of the story; this could help children build bridges to more mature cognitive and language abilities necessary for tackling reading and writing.
In this project we aim to develop a new technology of self-cutting paper system that generates the cutting shapes automatically based on the signal input. Our key novelty lies on embedding a grid of light-weight heating wires into normal paper during the paper making process. As a result, the augmented paper can automatically cut itself by activating different units in the grid of self-cutting mechanism. Therefore, we produce a safe and accurate self-cutting paper.
Unlike the existing works in interactive paper-based system which are pre-designed before the interaction, our technology allows users to define the shapes and the interactions of the physical paper crafts. This could be applied to not only entertainment such as story-telling, gaming, but also designing and rapid prototyping in architecture, fashion and other industrial design.
As the initial implementation, we are trying to use magnetic-based wireless power transfer to provide power to the heating elements which are attached to the paper, so the paper can cut by itself instead of connecting to various wires with power supply. Patterns of receiving coils will be made by the ni-chrome heating element into different shapes, and embedded in the paper. By controlling different frequencies and powers in the source coil, we hope to trigger different receiving coils to heat up in different positions, further to form different shapes of cut-up according to user input. In current progress, simulation of the electromagnetic effects has been done and significant results are attained.
Our technology could be applied to various areas, such as cultural education, story-telling, gaming, designing and so on, especially for children. As an example, children could use the system as an educational and creative tool to create their own landscapes and stories in paper to play with by inputting their drawing or pictures, and our technology combining with visionbased technology could generate the physical cut-out paper landscapes automatically. This provides high freedom for children to expand and experiment their imagination and creativity.
In addition, in the scenario of cultural computing, with our technology children can easily create their own traditional Chinese art of paper-cutting. Our system could automatically convert their drawing into the style of Chinese paper-cutting digitally and physically, further stimulating children’s interest to preserve this disappearing traditional art.
(e) Petimo: Social Networking Robot for Children
Nowadays, the world is crowded with all kinds of media and new technology. Social websites and online communication are becoming an integral trend especially among young children for connecting friends and developing social networks. However, psychologists have theorized about the influence of online communication toward children’s development and have warned about the dangers of exploitative relationships through Internet. To protect children from the potential risks in social networks and the virtual world, “Petimo” had been developed as an interactive robotic toy which empathizes that physical interaction applied in online social networking platform. Petimo can help children learn how to connect to their friends safely on Internet and increases the enjoyment of online communication.
An experience like no other, Petimo responds to touch and movement. Its outer finish is an adorable plush toy and its small, round physical shape evokes a warm and tender feeling likened to carrying a baby animal. Petimo has already been selected by an international review panel as one of the best three submissions for the Como for Children (C4C) design competition held at The 8th International Conference on Interaction Design and Children (IDC 2009) during its debut in Europe. Recently, Petimo won the first prize in an international innovation competition in Milan, Italy (Apr 22nd 2010). The prize is InventiON: concorso di idee per inventori InventiON: competition of ideas for inventors http://www.innovationcircus.it/2009/ .
Petimo won the first prize in the ICT (information and communication technologies) track. Petimo comes complete with a three-dimensional (3D) virtual world that features social networking capabilities as well. The virtual characters in Petimo-World are linked to the physical attributes of the robotic toy, resembling virtual characters interacting with each other accordingly. Parents can also supervise their children’s online behavior in a friendly and secure manner through its parental authentication module which enhances not only children’s online security but also the relationship between children and parents.
Special considerations also ensure that the physical properties and robotic interactions are wellsuited to children. For instance, Petimo has the enhanced capability to detect another robot placed together in a given proximity. This physical meet between two Petimos s a first that proves to be an even safer way for children to meet friends virtually, and also lends a virtual world feel to a child’s social networking experience.
Other features, such as a squeeze, can generate various emotions and gifts in audible and visual expressions as well. A specially-designed interface presents shapes and sizes that are appealing, simple and easy to use. The emotions and virtual gift transactions are also designed for a child’s easy understanding. Petimo, together with Petimo-World, can also be a mediator in cultivating positive social behavior and better familiarization to computing environments among young growing minds.
However, the concomitant risk of the fast-paced development of SNS could be trust and privacy concern with SNSs, as the sharing and chatting features may result in personal information being revealed, either consciously or unconsciously, especially for children who might be less alert to these safety issues than adults. Our current research focus is to explore the aforementioned risks and find out parents’ as well as children’s perception of the Petimo.
According to children’s and parents’ perspectives, we conducted the user study considering two factors each from two parties and we separately used quantitative and qualitative approaches to explore our hypothesis for above two factors.
As Petimo can create more intended communication among children, as well as between children and parents, our first and most important research approach is to explore “Communicative enjoyment with physical interaction”. The physical interaction can enhance the enjoyment and satisfaction of online communication for children. For this purpose, our intended participants are children from 7 – 10 years old as we designed the system for that age range and we collected quantitative data to prove our hypothesis about communicative enjoyment gaining from physical interaction while children engage with SNS. Then with the parents of those participated children, we conducted a qualitative user study to explore safety concern which is addressed by Petimo through physical interaction. Because Petimo is a more secure platform for young generation to engage in social activities online. The safety factor is crucial both in the physical and virtual world. Parent can feel released and safe about children when they communicate in Petimo online social networking system.
With 47 participants, 26 children and 21 parents, a formal user study is conducted in Singapore by following the main principle of children’s development theory. The study evaluated the system and found that the physical interaction is capable of enhancing communicative enjoyment between children, and also improving the safety factor for parents to feel more released about children online social activity. Compared with only using Petimo world (virtual world), children can receive more pleasure and satisfaction in communication by using both Petimo world and Petimo robot (physical interaction). Many children replied the system with physical interaction could not only improve the fun factors, especially in physical friends adding feature, in Petimo system, but also increase their intention to buy and use Petimo in their real daily life.
For the parents’ side, many participants reported when observing their children playing with Petimo, they think online SNS environment with physical interaction is safer and capable of solving the problems they concerned. One parent, who also has a teenage daughter, told us she has been worried about her elder child using Facebook, but after observing our system, she thought Petimo provided a very good entry platform for children who start engaging in online social network. It could educate children how to make friends and maintain their friendships in a secured way via Internet. From other comments received, some participants suggested that the system could become more secured if we limit the free chat function in Petimo system. There is one parent who works in an online game company who suggest us to change the current chat feature into fixed messages, thus children can still express their feeling but in a more protected way. We will address these issues in our next version of the system. In the future, Petimo is aiming to be mass manufactured and commercialized as a real product. Petimo was designed as a commercially oriented product / service, and it is adequate for dealing with demand changes and large product variety without needing reconfiguration. The research team is planning to actively work with VC’s in the future including VCs in Singapore. With the high expectation, Petimo will assume an important role in creating a safer playing environment. Petimo will dramatically change the younger generation’s tendency of being disconnected from family and loved ones by bridging the gaps of existing social network security issues and acting as a powerful means to support a child’s safe path toward a secured and personally enriching social networking experience.
4. Where does most of the funding for managing your lab come from?
Major funding for our laboratory has come from the Interactive and Digital Media Office of the Media Development Authority in Singapore, Ministry of Defence in Singapore, Defence Science and Technology Agency in Singapore, and companies including NEC, Nike, Thymos. Our lab has spun off three companies including MXR Corporation, Brooklyn Media, and Real Space.
5. How do you distribute the budget for a project? (For example: research expenses, operating expenses, personnel expenses...etc.)
This amount varies from project to project and also according to the requirements of the funding agencies. However normally the manpower costs (researcher salaries, student Ph.D scholarships) are more than 60% of the total budget. We then expend for equipments and consumables to support the lab research projects.
6. If your budget comes from a university or a company, what are the advantages and disadvantages of this type of funding?
There are advantages and disadvantages for both government funding and company funding. For government funding the application process and ongoing administration is often quite laborious and time consuming. However the benefit is that often we can undertake much more blue sky research and openly publish our work. Company funding is more flexible and it is always interesting to work with private industry. However, sometimes the focus of the company may be a little short term and require permission to publish our results.
7. If your lab has experienced cases of activities on the border of arts [and] industries, were there any difficulties in completing the projects? Please explain your difficulties to us and if you had successful results despite the hardships.
Employing practitioners from fields such as engineering, design, fine art, physiology, social sciences and more, the Mixed Reality Lab exists on the cusp of various fields, where projects exist between the lines of traditional knowledge.
Even though such multi-talented, multi-disciplinary teams are essential for many of the projects and collaborations in the lab, there are still great difficulties when engaging internally between collaborators, and externally to community and industry partners. Internally, each and every researcher in the lab is unique. People are experts in specific fields, come from different cultures and have varying working methodologies. On a professional level, designers and engineers can use terminology that is specific to the jargon of their respective fields. What one term means to one group of professionals could mean something completely different to another. To have a common language, in which all members of the lab can communicate their ideas with is key.
Because of this need, not only do projects cross the boundaries of various fields, but its conceptualizers and implementors must also do the same. Lab members are encouraged to learn the practice of their fellow members and it isn't foreign to see engineers using designdriven research or designers tinkering in order to learn electronics and soldering. Understanding the language and knowledge of their fellows is key in order to communicate and collaborate
This sharing helps researchers engage people of all backgrounds, not only because of a mutual understanding of the skills involved that are needed for every project, but also because of the sense of mutual respect and understanding for each other as individuals. This becomes doubly important when collaborating with industry partners and in ideal situations, it also communicates to our partners that lab teams are knowledgable and cohesive. Of course, difficulties still arise when people of differing objectives collaborate, which is almost always the case when artistic/academic research and industry come together.
An example of this can be found in a recent project conducted in the lab, in which the lab was commissioned to develop new technologies and interaction methodologies between people in remote locations. Because of the lab's drive for quantum innovation, many of the ideas developed between lab practitioners were far-forward thinking, and even inconceivable to industry partners who have a contemporary understanding of the topic chosen in relation to the market they target.
What lab researchers want to pursue often differs greatly to what industry partners wanted to see, and it took much deliberation and compromise between both sides in order to come to an agreement. These outcomes can be somewhat troubling, as neither side of the collaboration (lab and industry partner) feels satisfied with the products finally agreed upon to develop.
Luckily for this collaboration, the projects proposed and implemented were not too far off the mark of expectations from the lab's perspective, but this of course could have turned horribly wrong.
In the end what saved the project was a clear realization of the expectations of all parties. By collectively having a working knowledge in many fields, fruitful conversations between lab members and industry partners could be achieved. The lab was finally able to communicate the importance of the research conducted by relating it to the expectations and aspirations of the industry partner. It is not only the ability to understand what an industry partner wants now, but to more importantly communicate what an industry partner could want for the future. Our job is to express the possibilities of their business tomorrow.
8. Please tell us of the successes your lab has had in cooperating with other organizations (in all forms).
We have been greatly enriched by collaborating with other universities, professors, and researchers as well as organizations such as art museums, science centers, schools, and NGO’s. Some good examples are collaboration projects and exhibitions we have undertaken with Ars Electronic Museum, Singapore Science Center, Deajeon Municipal Museum of Art Korea, Singapore Art Museum. These have provided a different perspective to our research as we have been able to obtain excellent feedback from society and the community. We have worked with Hougang Primary School in Singapore to set up a Mixed Reality Classroom. This was a unique perspective on focusing our research for primary school children. We are also currently working with the Youth Task Force of the World Economic Forum Young Global Leaders to provide new frameworks and methods to increase the creativity and innovation in children, particularly in the developing world. We also have been enriched by intern students from all over the world, and from local schools in Singapore, who join our lab and specific projects.
9. Are there any important factors to creating a stimulating and productive research/work
Creative and innovative research is the fruit of a myriad of factors that are stirred together and simmered. The key element here is the people themselves, when people from a range of disciplines such as engineering, social science, art, design, business and others come together and work as a team, they bring with them their individual knowledge and personal experience in the field. When knowledge is shared, ideas begin to build on each other and the result is research and solutions that are unique and more complete, tackled from every angle. Each project in our lab brings together people not only from different academic background but also cultural backgrounds and we value the rich and distinctive perspective each one brings to the table.
Brainstorming sessions can be the most precious time spent by any research group. An open and comfortable environment that is built on trust will encourage people to share all kinds of ideas without any fear or inhibition and this free flow of ideas is what later spins into innovativeresearch. The environment also needs to be one that encourages play, laughter, fuel curiosity and challenge each person to push themselves and learn new ways of thinking, looking, making and doing things. Working with ones hands, pulling things apart and making new things is also a way for ideas to flow and build. This happens when we are willing to step out of our comfort zones and take a few risks. When you give people the freedom to choose their directions and ideas that drive their passion, this will naturally lend itself to inspiring and ground breaking works.
All of the above needs to be tied together with times of reflection - time to absorb knowledge, to churn it around in your mind and to let the connections be made. This is where the magic happens. Ideas and thoughts need time to brew, it will not happened when one is overworked, stressed and not rested enough. As obvious as this may be, this is one factor often people overlook, leading to many hours spent on work but very little creative work or research
It is a fine balance of all these factors that gets the creative juices flowing and when a dynamic group of people who have a passion to push boundaries, create new knowledge and have ideas dripping out of them you cannot predict the results but it will help create results and ideas that will change the world for the better.
10. Please tell us of your next project or plan.
In general our vision in the laboratory is to invent, examine, and create new forms of media and communication that will benefit society and the creative economy. We have a particularly interest in family and emotional communication, and the next generation of children’s learning, innovation, creativity and play.
Just like previous seismic shifts in civilization, the net age has produced a massive change in civilization. The main effect of this shift has been the instant, global, and constant communication, and so in some way this has been a revolutionary change of human communication. What this has meant is that the net generation grow up with new forms of play and communication from previous youth. Their entertainment is immersive, fantastic, and can be played together with thousands or millions of people around the world. Their communication is instant, from many multiple sources simultaneously, constant, and global. In some way there has been an end of solitude (which may also have negative consequences). Now, when the net generation goes to school, they find it totally different, and more and more irrelevant from their daily lives. They are used to constant, immersive, simultaneous multiple source communication.
Sitting in a traditional classroom, and being told not to use their mobile phones or send messages must be totally boring and frustrating to them. Having one source of information from the teacher, must seem so slow. They may feel they wish they could press a fast forward button to the teacher.
Not only for the young net generation, but all of society has radically changed. Grandparents are playing games on Nintendo Wii or DS, and parents are using Twitter. What this means is that in general we must understand the new media and the new communication in order to enrich learning and education that is relevant, and also to allow work, play, culture, family life, and elderly care more relevant and enriching.