The Forbes website published an article introducing Facebook's efforts in augmented reality. Facebook executives said that if the company still has a significant position in the industry in the next 10 or 20 years, it must achieve an outstanding performance in augmented reality. The following is the original content:
In January 2014, Mark · Zuckerberg first tried the virtual reality (VR) headset Oculus Rift. Put on this brick-like device, he from the office "stellar" to the ruins of a medieval castle, surrounded by thick snowflakes. He immediately decided that VR will one day become a mainstream computing platform. Two months later, Facebook spent $2 billion to acquire Oculus.
But there is also a technology called Augmented Reality (AR), which emerges later than VR. It can be used directly on smartphones to superimpose background information or special digital effects onto the physical world. Zuckerberg ordered engineers to start developing AR and VR at the same time. Because their underlying technologies are the same, a two-pronged approach is logical.
Mike Schroepfer, Facebook’s chief technology officer, said that AR and VR ranked in the top three of Facebook’s technology priorities, with the other two being connectivity and artificial intelligence (AI). Within Facebook, hundreds of engineers are working on basic technologies such as computer vision to enable mobile phones to track facial movements in real time, identify coffee cups, or recommend images that match the background. AI is very important for AR, and Facebook employees often refer to in-app cameras as "AI cameras". The company has put "very much money" on this, "Skoropf said.
Today, Facebook is competing with Apple and Google, and it can be said that it still competes with Snap to some extent, hoping to become the platform of choice for AR developers. Although the technology itself is still in its infancy, it has quickly become popular. This proves that Zuckerberg's instincts are correct and that VR is still an embarrassing technology. Its appeal is mainly confined to hard. The nuclear player circle, and AR can be popularized to the public. The key advantage of the AR is that it does not require expensive, cumbersome helmets, and users only need to have a mobile phone. More than a quarter of the world’s people currently own mobile phones.
Facebook application machine learning leader Jacque · Candela (Joaquin Candela) said: "The number of mobile phone owners is much more than the VR helmet owners. It is obvious that focusing on the AR. ”
And early evidence suggests that AR can attract consumers and sometimes radically change the way they interact with mobile phones. Some AR apps are extremely popular, such as the puppy mask on Snapchat and the Elven Pokémon treasure hunt game. The situation is clear: The next major phase of mixed reality is AR, not VR.
However, the reason why technology giants invest heavily in AR is that AR applications are far more than social media, games, and bizarre special effects. It can be used in many areas from navigation to e-commerce to power online shopping. IKEA's app can already help shoppers virtually place furniture in their home. Some pharmaceutical companies are also using AR to show real-time information about drugs. Modern companies use AR applications to help consumers understand some of the features of the car. AR can also provide tutorials for customers along with chat robots. Tom Meyer, manager of the Facebook smart phone team, said: "Smart phones can be magic eyes for you to watch the world." ”
Inside Facebook, engineers and executives admit it's important. If you do not take the wrong direction on the AR, the company's app usage may decline. Users’ loyalty to social applications may be fleeting, and Facebook is competing with Snapchat for young users. Which platform has the most attractive features, the most engaging communication methods (especially through pictures and video communication), and the evolving experience, users will quickly shift to that platform.
Facebook launched its own AR feature a few years after Snapchat launched AR. In the Internet age, success or failure may be determined one step later. Google’s social media product G+ is an example. However, because Facebook has a strong AI (which can support larger advanced effects) and excellent product design, it quickly caught up with Snapchat, preventing the continued growth of this competitor.
The success of AR can bring great returns to Facebook. The user’s activity and duration of stay in the company’s app contributed to the growth of Facebook’s advertising business, helping it generate $26.9 billion in revenue in 2016. The AR effect prompts users to spend more and more total time on Facebook, encourages users to send messages more frequently, spend more time viewing friends' posts and making their own posts. Facebook users spend an average of about 50 minutes a day on Instagram and Messenger, and Facebook needs to continue to introduce new products to attract attention and promote "image-intensive" communication methods to maintain high standards in competition. The occupies a dominant position, especially in the company's efforts to combat the "headline party" and false news.
The popularity of special effects and filters in Facebook applications marks an important step in the company’s transition from a still image to a “camera of the future”—— a charming, AI-driven effect spread across friends and family in. Puppy effects may seem boring, but it shows Facebook's direction on the AR. Chris Cox, Facebook’s chief product officer, said that “delegative communication” is just as important as moving from desktop to mobile.
“If we hope that Facebook will still play an important role in the industry in the next 10 or 20 years, we must achieve excellence in AR. "Skoropf said.
Facebook is now the world’s largest social AR ecosystem, but the first company to make social AR popular is Snapchat. Until more than a year ago, Snapchat's AR functionality was far ahead of Facebook. However, due to Zuckerberg's early anticipation that AR will one day become a mainstream communication tool, Facebook has been focusing on developing its behind-the-scenes core technology.
Facebook has been closely watching Snapchat's move, and it is also very clear about the trends on its own app. Posts on Facebook are increasingly turning to photos, GIFs, and videos, and texts are becoming scarce. The evidence clearly shows that users want to express themselves through images and animation on Facebook, and they must do this in real time.
It is unavoidable to increase investment in image sharing. When Facebook introduced real-time video and noticed a self-portrait effect called Masquerade (MSQRD), the company's product roadmap took a major turn. MSQRD had already achieved success in Eastern Europe at the time and triggered an upsurge in the United States with a total of 16 million users. The founder Eugen Zatepyakin spent nearly three years creating 3D graphics rendering technology that made MSQRD's fancy effects tools look almost the same as Snapchat's. Facebook executives were fascinated by it.
Facebook acquired the startup in March 2016 and the transaction price was not disclosed. After that, Facebook began to increase its efforts to make up for the waste of time. Soon after, Zuckerberg published a video of his own use of MSQRD Iron Man's special effects on social networks, and quietly launched a major internal project: to quickly build technical support for more complex in-app cameras.
Breakthrough on AI
Facebook has been working on the development of AI. However, until Zuckerberg called for the formation of a dedicated "camera team" in the summer of 2016, the company's AR efforts became apparent. The "camera team" was originally a member of the AI engineers and researchers of the Facebook application machine learning team.
Over the next year and a half, the "camera team" has expanded to more than 100 people, including designers from Hollywood and game companies, who work collaboratively with the camera product managers of companies such as Messenger and Instagram to help them. Publish features and quickly iterate. From advancing underlying visual identity and deep learning techniques to organizing external user groups to try out AR tools, the team's scope of work is quite extensive.
For example, in a user group, someone uses an animation effect (expression pack) to wish his wife a happy birthday, which is easier and faster than using a textual language.
"He used the tools we provided to create a very intimate and cuddly experience. The AR tool provided him with a convenient form of expression." When people want to express their emotions in a stupid way, these tools make them more confident. ” A team member said.
Shortly after the camera team was established, Pokémon Dream had great success, further demonstrating that AR can attract mass consumers. After 19 days in July 2016, the game attracted 50 million players. In order to use smartphones to search for AR characters, players often walk for miles. Zuckerberg himself is also a fan of this game. This game proves to Facebook executives that AR is not limited to messaging applications like Snapchat.
At the same time, the engineers of the Facebook camera team are building internal AI processing software, and they hope this will be a support for future AR results. Competitors like Snapchat rely on external servers to provide AR functionality, reducing speed and complexity, but Facebook is trying to develop a system for handling AI directly on smartphones, which was later called Caffe2Go. Facebook hopes to build the infrastructure first, ensuring that it can scale seamlessly, and when it can provide better rendering, face tracking, and speed than Snapchat, it will launch AR effects.
While building Caffe2Go, the Facebook product team tried AR tools with the help of MSQRD. In August 2016, before the opening of the Summer Olympic Games, Facebook conducted some tests that imitated Snapchat style. It also launched "Stories" on Instagram, imitating Snapchat's signature feature that made posts disappear.
That autumn, Caffe2Go was completed. It is the first system that can handle real-time capture and analysis of pixels by using AI directly on the smartphone. Facebook tested Caffe2Go in the fall of 2016, allowing users to convert photos or videos into the style of Picasso or Van Gogh artists.
At the annual F8 Developer Conference held at Facebook in April 2017, Zuckerberg announced the launch of the "camera platform" to allow developers to build AR functionality on Facebook. Zuckerberg believes that in the end this technology will release people's creativity and create more works of art.
This is Zuckerberg's first use of Facebook in-app camera as the core of the platform's communication methods, compared with the self-proclaimed "camera company" Snapchat confrontation. Although AR glasses may be the first wearable AR device, Zuckerberg predicts that people will begin to enjoy the golden age of AR on smartphones.
Since Facebook began launching AR special effects at the end of 2016, Snapchat’s growth has stagnated. Only about 190 million people use Snapchat every day, and 500 million people use Facebook's Instagram every day.
Although AR's largest consumer use is on social media, almost all tech giants are building AR functionality for their own developer ecosystems and products, and most developers do not yet know how to use AR. Two months after Facebook launched its AR Camera Platform, Apple introduced its own developer toolkit, ARKit, for iOS 11. Developers and marketers can use it to easily integrate AR into their existing applications. . Research company Forrester's data shows that about 400 million devices are compatible with Apple's ARKit. In September last year, Apple introduced the "Animoji" feature for iMessage, which uses face recognition on iPhone X to allow people to create emoticons using their own facial expressions.
However, Google may have been the longest-running tech company on the AR front, and as early as 2013 it introduced Google Glass. This kind of smart device is highly praised by technology enthusiasts, but consumers are less concerned about privacy because of concerns about privacy. In 2016, Google launched an AR platform called Tango that uses depth sensors to map indoor space, but it is only compatible with a few devices. To expand Tango’s capabilities to more mobile phones without the need for these phones to add cameras and sensors, Google launched ARCore in August, similar to Apple’s ARKit. This developer toolkit targets existing and future Android devices such as the Samsung Galaxy 8.
Like Facebook, Google naturally is interested in using AR for search. In May, Google announced the launch of the Lens &rdquo function, a computer vision tool that can classify albums in Google Photos. Pixel and Pixel 2 users can now point the camera at an object such as a storefront to get relevant information in real time. One of the benefits of Apple and Google owning their own operating systems is that developers can build large numbers of AR applications on these operating systems. Compared with Apple's Google, Facebook has not yet successfully launched hardware products.
Forrester analyst Thomas & middot; Thomas Husson said: "Facebook's camera platform can help developers and marketers attract more and more audiences. But the limitation of Facebook is that it cannot control the hardware. The truly amazing AR experience requires the cooperation of software and hardware. ”
Although Facebook, Apple, and Google have all focused on smart phones, Microsoft has been focusing on business customers for wearable devices. The company launched Hololens in 2016, which is a $3000 VR helmet. Hololens uses a Windows Mixed Reality operating system that is also available to other VR and AR hardware manufacturers. However, it may take several years for the cost of an AR device to drop to the level of the people and attract the mass market.
Facebook may never have its own operating system or AR device (although it is trying), but it does have an important unique asset: First, it is covered by any other social or messaging application such as Snap, Kakao, Line, and WeChat. The users are more (and therefore have a lot of data about people's interests, which is very useful for personalizing AR effects). Second, Facebook has one of the largest computer vision teams to help it create nice features. Even if Facebook will never launch its own popular AR hardware, it will still be one of the largest AR content ecosystems in the coming years.
The next trend in advertising
At the end of the seventh quarter of the popular episode "Game of Thrones" in July last year, more than 1 million people created their own "night king" on Facebook, and sent it to friends. This face tracking effect can be adjusted in real time based on the user's face.
Importantly, the producer of this special effect is HBO, not Facebook. It quickly became one of the most successful AR ads on Facebook. This shows that high-quality AR effects can become advertisements that people send to their friends. As shown in the "night king" effects, AR is tempting for many marketers because it provides a new way for smartphone owners to stay longer. Facebook is positioning itself as the default platform for AR marketing campaigns, and its core is to encourage users to play AR games with friends and post AR selfies.
Only about 5% of marketers use AR technology. A recent Forrester study showed that 17% of marketers plan to use the technology this year, and that AR is more important to marketers than VR for at least the next three years.
It is still in the exploratory stage
As AR extends from smart phones to wearable devices, this technology may become a permanent enhancement to the human senses, and may provide search bar-like functionality based on our location, interests, and social networks. However, AR's use in consumption is still a new thing. In the next few years, major tech giants will be busy improving their core technologies and building ARs for smartphones, and Facebook will focus on the advantages of text messaging and personal expression. Over time, AR may make Facebook's dynamic news more immersive, video richer, more interactive, although the details are still difficult to say.
Facebook needs to constantly improve its own AI, improve speed and accuracy. For example, when identifying an object in a video, understand how it is pixelated when viewed from different angles, and understand how to map the relationships between objects in the scene. In order to attract users in developing markets, Facebook's AR features also need to be compatible with some weaker cellular networks and some older phones.
"We are still in the basic exploration phase and we are still building basic technology," Facebook's Candela said.
Skrotop said that Facebook is "investing heavily in hardware" to support AR and build more social tools. And there are other benefits to improving the underlying AI, such as helping to process spam and inappropriate content through better visual recognition and language understanding tools, and driving game and robotics.
Candela said: "From the social infrastructure we have built, the technology we promote, you may find it hard to imagine how to apply it. But I'm pretty sure that this will unlock what we can't imagine today. A meaningful AR experience will be very social. ”