Posts in category: Facebook AI

As AR and VR devices become a bigger part of how we work and play, how do we maintain seamless social connection between real and virtual worlds? In other words, how do we maintain “social co-presence” in shared spaces among people who may or may not be involved in the same AR/VR experience?

This year at SIGGRAPH, Facebook Reality Labs (FRL) Research will present a new concept for social co-presence with virtual reality headsets: reverse passthrough VR, led by research scientist Nathan Matsuda. Put simply, reverse passthrough is an experimental VR research demo that allows the eyes of someone wearing a headset to be seen by the outside world. This is in contrast to what Quest headsets can do today with Passthrough+ and the experimental Passthrough API, which use externally facing cameras to help users easily see their external surroundings while they’re wearing the headset.

Over the years, we’ve made strides in enabling Passthrough features for Oculus for consumers and developers to explore. In fact, the idea for this experimental reverse passthrough research occurred to Matsuda after he spent a day in the office wearing a Quest headset with Passthrough, thinking through how to make mixed reality environments more seamless for social and professional settings. Wearing the headset with Passthrough, he could see his colleagues and the room around him just fine. But his colleagues couldn’t see him without an external display. Every time he attempted to speak to someone, they remarked how strange it was that he wasn’t able to make eye contact. So Matsuda posed the question: What if you could see his eyes — would that add something to the social dynamic?

When Matsuda first demonstrated reverse passthrough for FRL Chief Scientist Michael Abrash in 2019, Abrash was unconvinced about the utility of this work. In the demo, Matsuda wore a custom-built Rift S headset with a 3D display mounted to the front. On the screen, a floating 3D image of Matsuda’s face, crudely rendered from a game engine, re-created his eye gaze using signals from a pair of eye-tracking cameras inside the headset.

Research scientist Nathan Matsuda wears an early reverse passthrough prototype with 2D outward-facing displays. Right: The first fully functional reverse passthrough demo using 3D light field displays.

“My first reaction was that it was kind of a goofy idea, a novelty at best,” said Abrash. “But I don’t tell researchers what to do, because you don’t get innovation without freedom to try new things, and that’s a good thing, because now it’s clearly a unique idea with genuine promise.”

Nearly two years after the initial demo, the 3D display technology and research prototype have evolved significantly, featuring purpose-built optics, electronics, software, and a range of supporting technologies to capture and depict more realistic 3D faces. This progress is promising, but this research is clearly still experimental: Tethered by many cables, it’s far from a standalone headset, and the eye and facial renderings are not yet completely lifelike. However, it is a research prototype designed in the spirit of FRL Research’s core ethos to run with far-flung concepts that may seem a bit outlandish. While this work is nowhere near a product roadmap, it does offer a glimpse into how reverse passthrough could be used in collaborative spaces of the future — both real and virtual.

Left: A VR headset with the external display disabled, representing the current state of the art. No gaze cues are visible through the opaque headset enclosure. Middle: A VR headset with outward-facing 2D displays, as proposed in prior academic works[1][2][3][4]. Some gaze cues are visible, but the incorrect perspective limits the viewer’s ability to discern gaze direction. Right: Our recent prototype uses 3D reverse passthrough displays, showing correct perspective for multiple external viewers.

Reverse passthrough

The essential component in a reverse passthrough headset is the externally facing 3D display. You could simply put a 2D display on the front of the headset and show a flat projection of the user’s face on it, but the offset from the user’s actual face to the front of the headset makes for a visually jarring, unnatural effect that breaks any hope of reading correct eye contact. As the research prototype evolved, it was clear that a 3D display was a better direction, as it would allow the user’s eyes and face to appear at the correct position in space on the front of the headset. This depiction helps maintain alignment as external viewers move in relation to the 3D display.

There are several established ways to display 3D images. For this research, we used a microlens-array light field display because it’s thin, simple to construct, and based on existing consumer LCD technology. These displays use a tiny grid of lenses that send light from different LCD pixels out in different directions, with the effect that an observer sees a different image when looking at the display from different directions. The perspective of the images shift naturally so that any number of people in the room can look at the light field display and see the correct perspective for their location.

As with any early stage research prototype, this hardware still carries significant limitations: First, the viewing angle can’t be too severe, and second, the prototype can only show objects in sharp focus that are within a few centimeters of the physical screen surface. Conversations take place face-to-face, which naturally limits reverse passthrough viewing angles. And the wearer’s face is only a few centimeters from the physical screen surface, so the technology works well for this case — and will work even better if VR headsets continue to shrink in size, using methods such as holographic optics.

Building the research prototype

FRL researchers used a Rift S for early explorations of reverse passthrough. As the concept evolved, the team began iterating on Half Dome 2 to build the research prototype presented this year at SIGGRAPH. Stripping down the headset to the bare display pod, mechanical research engineer Joel Hegland provided a roughly 50-millimeter-thick VR headset to serve as a base for the latest reverse passthrough demo. Then, optical scientist Brian Wheelwright designed a microlens array to be fitted in front.

The resulting headset contains two display pods that are mirror images of each other. They contain an LCD panel and lens for the base VR display. A ring of infrared LEDs illuminates the part of the face covered by the pod. A mirror that is reflective only for infrared light sits between the lens and screen, so that a pair of infrared cameras can view the eye from nearly head-on. Doing all this in the invisible infrared band keeps the eye imaging system from distracting the user from the VR display itself. Then the front of the pod has another LCD with the microlens array.

Left: A cutaway view of one of the prototype display pods. Right: The prototype display pod with driver electronics, prior to installation in the full headset prototype.

Imaging eyes and faces in 3D

Producing the interleaved 3D images to show on the light field display presented a significant challenge in itself. For this research prototype, Matsuda and team opted to use a stereo camera pair to produce a surface model of the face, then projected the views of the eye onto that surface. While the resulting projected eyes and face are not lifelike, this is just a short-term solution to pave the way for future development.

FRL’s Codec Avatars research points toward the next generation of this imaging. Codec Avatars are realistic representations of the human face, expressions, voice, and body that, via deep learning, can be driven from a compact set of measurements taken inside a VR headset in real time. These virtual avatars should be much more effective for reverse passthrough, allowing for a unified system of facial representation that works whether the viewer is local or remote.

Shown below, a short video depicts a Codec Avatar from our Pittsburgh lab running on the prototype reverse passthrough headset. These images, and their motion over time, appear much more lifelike than those captured using the current stereo camera method, indicating the sort of improvements that such a system could provide while working in tandem with remote telepresence systems.

The reverse passthrough prototype displaying a high-fidelity Codec Avatar facial reconstruction.

A path toward social co-presence in VR

Totally immersive VR and AR glasses with a display are fundamentally different technologies that will likely end up serving different users in different scenarios in the long term. There will be situations where people will need the true transparent optics of AR glasses, and others where people will prefer the image quality and immersion of VR. Facebook Reality Labs Research, under Michael Abrash’s direction, has cast a wide net when probing new technical concepts in order to move the ball forward across both of these display architectures. Fully exploring this space will ensure that the lab has a grasp on the full range of possibilities — and limitations — for future AR/VR devices, and eventually put those findings into practice in a way that supports human-computer interaction for the most people in the most places.

Reverse passthrough is representative of this sort of work — an example of how ideas from around the lab are pushing the utility of VR headsets forward. Later this year, we’ll give a more holistic update on our display systems research and show how all this work — from varifocal, holographic optics, eye tracking, and distortion correction to reverse passthrough — is coming together to help us pass what we call the Visual Turing Test in VR.

Ultimately, these innovations and more will come together to create VR headsets that are compact, light, and all-day wearable; that mix high-quality virtual images with high-quality real-world images; and that let you be socially present with anyone in the world, whether they’re on the other side of the planet or standing next to you. Making that happen is our goal at Facebook Reality Labs Research.

---

[1] Liwei Chan and Kouta Minamizawa. 2017. FrontFace: Facilitating Communication between HMD Users and Outsiders Using Front-Facing-Screen HMDs

[2] Kana Misawa and Jun Rekimoto. 2015. ChameleonMask: Embodied Physical and Social Telepresence using Human Surrogates

[3] Christian Mai, Lukas Rambold, and Mohamed Khamis. 2017. TransparentHMD: Revealing the HMD User’s Face to Bystanders

[4] Jan Gugenheimer, Christian Mai, Mark McGill, Julie Williamson, Frank Steinicke, and Ken Perlin. 2019. Challenges Using Head-Mounted Displays in Shared and Social Spaces

The post Display systems research: Reverse passthrough VR appeared first on Facebook Research.

Read More

This work was a collaboration with Martin Jankowiak (Broad Institute of Harvard and MIT).

What the research is:

Sparse axis-aligned subspace Bayesian optimization (SAASBO) is a new sample-efficient method for expensive-to-evaluate black-box optimization. Bayesian optimization (BO) is a popular approach to black-box optimization, with machine learning (ML) hyperparameter tuning being a popular application. While they’ve had great success on low-dimensional problems with no more than 20 tunable parameters, most BO methods perform poorly on problems with hundreds of tunable parameters when a small evaluation budget is available.

In this work, we propose a new sample-efficient BO method that shows compelling performance on black-box optimization problems with as many as 388 tunable parameters. In particular, SAASBO has shown to perform well on challenging real-world problems where other BO methods struggle. Our main contribution is a new Gaussian process (GP) model that is more suitable for high-dimensional search spaces. We propose a sparsity-inducing function prior that results in a GP model that quickly identifies the most important tunable parameters. We find that our SAAS model avoids overfitting in high-dimensional spaces and enables sample-efficient high-dimensional BO.

SAASBO has already had several use cases across Facebook. For example, we used it for multiobjective Bayesian optimization for neural architecture search where we are interested in exploring the trade-off between model accuracy and on-device latency. As we show in another blog post, the SAAS model achieves much better model fits than a standard GP model for both the accuracy and on-device latency objectives. In addition, the SAAS model has also shown encouraging results for modeling the outcomes of online A/B tests, where standard GP models sometimes have difficulties to achieve good fits.

How it works:

BO with hundreds of tunable parameters presents several challenges, many of which can be traced to the complexity of high-dimensional spaces. A common behavior of standard BO algorithms in high-dimensional spaces is that they tend to prefer highly uncertain points near the domain boundary. As this is usually where the GP model is the most uncertain, this is often a poor choice that leads to overexploration and results in poor optimization performance. Our SAAS model places sparse priors on the inverse lengthscales of the GP model combined with a global shrinkage that controls the overall model sparsity. This prior results in a model where the majority of dimensions are “turned off,” which avoids overfitting.

An appealing property of the SAAS priors is that they are adaptive. As we collect more data, we may gather evidence that additional parameters are important, which allows us to effectively “turn on” more dimensions. This is in contrast to a standard GP model with maximum likelihood estimation (MLE), which will generally exhibit non-negligible inverse lengthscales for most dimensions—since there is no mechanism regularizing the lengthscales—often resulting in drastic overfitting in high-dimensional settings. We rely on the No-Turn-U-Sampler (NUTS) for inference in the SAAS model as we found it to clearly outperform maximum a posteriori (MAP) estimation. In Figure 1, we compare the model fit using 50 training points and 100 test points for three different GP models on a 388 dimensional SVM problem. We see that the SAAS model provides well-calibrated out-of-sample predictions while a GP with MLE/NUTS with weak priors overfit and show poor out-of-sample performance.

Figure 1: We compare a GP fit with MLE (left), a GP with weak priors fit with NUTS (middle), and a GP with a SAAS prior fit with NUTS (right). In each figure, mean predictions are depicted with dots, while bars correspond to 95 percent confidence intervals.

Many approaches to high-dimensional BO try to reduce the effective dimensionality of the problem. For example, random projection methods like ALEBO and HeSBO work directly in a low-dimensional space, while a method like TuRBO constrains the domain over which the acquisition function is optimized. SAASBO works directly in the high-dimensional space and instead relies on a sparsity-inducing function prior to mitigate the curse of dimensionality. This provides several distinct advantages over existing methods. First, it preserves—and therefore can exploit—structure in the input domain, in contrast to methods that rely on random embeddings, which risk scrambling it. Second, it is adaptive and exhibits little sensitivity to its hyperparameters. Third, it can naturally accommodate both input and output constraints, in contrast with methods that rely on random embeddings, where input constraints are particularly challenging.

Why it matters:

Sample-efficient high-dimensional black-box optimization is an important problem, with ML hyperparameter tuning being a common application. In our recently published blog post on Bayesian multiobjective neural architecture search, we optimized a total of 24 hyperparameters, and the use of the SAAS model was crucial to achieve good performance. Because of the high cost involved with training large-scale ML models, we want to try as few hyperparameters as possible, which requires sample-efficiency of the black-box optimization method.

We find that SAASBO performs well on challenging real-world applications and outperforms state-of-the-art methods from Bayesian optimization. In Figure 2, we see that SAASBO outperforms other methods on a 100-dimensional rover trajectory planning problems, a 388-dimensional SVM ML hyperparameter tuning problem, and a 124-dimensional vehicle design problem.

Figure 2: For each method, we depict the mean value of the best value found at a given iteration (top row). For each method, we show the distribution over the final approximate minimum as a violin plot, with horizontal bars corresponding to 5 percent, 50 percent, and 95 percent quantiles (bottom row).

Read the full paper:

https://research.fb.com/publications/high-dimensional-bayesian-optimization-with-sparse-axis-aligned-subspaces/

View tutorial:

https://github.com/facebook/Ax/blob/master/tutorials/saasbo.ipynb

The post High-dimensional Bayesian optimization with sparsity-inducing priors appeared first on Facebook Research.

Read More

What the research is:

Facebook products run on a highly complex infrastructure system that consists of servers, network, back-end services, and client-facing software. Operating such systems at a high level of performance, reliability, and efficiency requires real-time monitoring, proactive failure detection, and prompt diagnostic of production issues. While a number of research and applications have addressed the need for monitoring the use of state-of-the-art anomaly detection, the diagnostics of root causes remains a largely manual and time-consuming process. Modern software systems can be so complex that unit/integration testing and error logs alone are not humanely tractable for root causing. Triaging an alert, for instance, would require manually examining a mixture of structured data (e.g., telemetry logging) and unstructured data (e.g., code changes, error messages).

The Infrastructure Data Science team at Facebook is developing a unified framework of algorithms, as a Python library, to tackle such challenges (see Figure 1). In this blog post, we illustrate applications of RCA from large-scale infrastructure systems, and discuss opportunities for applying statistics and data science to introduce new automation in this domain.

Figure 1. RCA methodologies and applications to infrastructure problems

How it works:

I. Attributing ML performance degradation to data set shift

Machine learning is an important part of Facebook products: It helps recommend content, connect new friends, and flag integrity violations. Feature shifts caused by corrupted training/inference data are a typical root cause of model performance degradations. We are investigating how to attribute a sudden change of model accuracy to the shifting data distributions. Machine learning models usually consume complex features, such as images, text, and high-dimensional embeddings as inputs. We apply statistical methods to perform changepoint detection on these high-dimensional features, and build black-box attribution models, agnostic of the original deep learning models, to attribute model performance degradation to feature and label shifts. See Figure 2 for an example of exposing shifted high-dimensional embedding features between two model training data sets. The methodology is also applicable to explaining accuracy degradations of an older model whose training data distribution differs from the inference data set.

Figure 2. An example of a sudden drastic data set shift in high-dimensional embedding features. Two-dimensional projections of the embeddings (using T-SNE) before and after the shift are visualized. This example, shown as an illustration using synthetic data, is similar to shifts observed in production settings.

II. Automatic diagnosis of key performance metric degradation

Infrastructure systems are monitored in real time, which generates a large amount of telemetry data. Diagnostic workflows usually start with drill-down data analysis, e.g., running analytical data queries to find which country, app, or device type shows the largest week-over-week reliability drop. Such insights could point the on-call engineer to the direction for further investigations. We experiment with dynamic programming algorithms that can automatically traverse the space of these subdimensions. We also try to fit a predictive model using the metrics and dimensions data set, and identify interesting dimensions by looking at feature importance. With the help of such tools, the time spent on repetitive analytical tasks is reduced.

Another diagnostic task is to examine what correlated telemetry metrics may have caused the key performance metric degradation. For instance, when latency of a service spikes, its owner may manually browse through the telemetry metrics of (sometimes a large number of) dependent services. Simple automations such as setting up anomaly detection for every metric can lead to noisy and false positive discoveries. A better approach, shown in Figure 3, is to learn from historical data about the temporal correlations between suspect metrics and the key performance metric, and tease out real root causes from spuriously correlated anomalies.

Figure 3. Methodology for evaluating and rank-ordering potential root-causing factors.

III. Event ranking and isolation

Many production issues are caused by internal changes to the software/infrastructure systems. Examples include code changes, configuration changes, and launching A/B tests for new features that affect a subset of users.

An ongoing research is to develop a model to isolate the changes that are potential root causes. As a first step, we use heuristic rules such as ranking based on time between code change and production issue. There is an opportunity to adopt more signals such as team, author, and code content to further reduce false positives and missing cases compared with the simple heuristic. A machine learning–based ranking model can effectively leverage such inputs. The limited amount of labeled data is a roadblock to automatically learning such rules. A possible solution is to explore a human-in-the-loop framework that iteratively collects subject-matter-expert feedback and adaptively updates the ranking model (see Figure 4).

Figure 4. A human-in-the-loop framework for blaming bad code changes.

At Facebook scale, there are numerous code/configuration/experimentation changes per day. Simply trying to rank order all of them cannot work. The ranking algorithm needs “prior” knowledge about the systems so as to narrow down the pool of suspect root-causing changes. For example, all the back-end services can be represented as a graph with edges representing how likely the degradation of one node can cause production issues of its neighbors. One example algorithm to build such a graph is to apply a deep neural network framework that represents the dynamic dependencies among a large number time series. Another possible direction is to apply causal graph inference models to discover the degree of dependencies among vertices. With the help of such prior knowledge, the isolation of bad changes can be achieved more effectively.

Why it matters:

Operating an efficient and reliable infrastructure is important to the success of Facebook products. While production issues would inevitably happen, quickly identifying root causes using data can expedite remediation and minimize the damage of such events. The proposed framework of algorithms will enable automated diagnosis using a mix of structured data (e.g., telemetry) and unstructured data (e.g., traces, code change events). The methodologies are developed in such a way that they can be generically applicable across different types of infrastructure systems. The algorithms, written as a Python library, can also be useful to the data science and software engineering community externally. Root cause analysis is an emerging space in data science that is at the intersection of existing areas such as data mining, supervised learning, and time series analysis.

The post Automating root cause analysis for infrastructure systems appeared first on Facebook Research.

Read More

What the research is:

We propose a method for sample-efficient optimization of the trade-offs between model accuracy and on-device prediction latency in deep neural networks.

Neural architecture search (NAS) aims to provide an automated framework that identifies the optimal architecture for a deep neural network machine learning model given an evaluation criterion such as model accuracy. The continuing trend toward deploying models on end user devices such as mobile phones has led to increased interest in optimizing multiple competing objectives in order to achieve an optimal balance between predictive performance and computational complexity (e.g., total number of flops), memory footprint, and latency of the model.

Existing NAS methods that rely on reinforcement learning and/or evolutionary strategies can incur prohibitively high computational costs because they require training and evaluating a large number of architectures. Many other approaches require integrating the optimization framework into the training and evaluation workflows, making it difficult to generalize to different production use-cases. In our work, we bridge these gaps by providing a NAS methodology that requires zero code change to a user’s training flow and can thus easily leverage existing large-scale training infrastructure while providing highly sample-efficient optimization of multiple competing objectives.

We leverage recent advances in multi-objective and high-dimensional Bayesian optimization (BO), a popular method for black-box optimization of computationally expensive functions. We demonstrate the utility of our method by optimizing the architecture and hyperparameters of a real-world natural language understanding model used at Facebook.

How it works:

NLU Use-Case

We focus on the specific problem of tuning the architecture and hyperparameters of an on-device natural language understanding (NLU) model that is commonly used by conversational agents found in most mobile devices and smart speakers. The primary objective of the NLU model is to understand the user’s semantic expression and to convert it into a structured decoupled representation that can be understood by downstream programs. The NLU model shown in Figure 1 is an encoder-decoder non-autoregressive (NAR) architecture based on the state-of-the-art span pointer formulation.

Figure 1: Non-autoregressive model architecture of the NLU semantic parsing

The NLU model serves as the first stage in conversational assistants and high accuracy is crucial for a positive user experience. Conversational assistants operate over the user’s language, potentially in privacy-sensitive situations such as when sending a message. For this reason, they generally run “on-device,” which comes at the cost of limited computational resources. Moreover, it is important that the model also achieves short on-device inference time (latency) to ensure a responsive user experience. While we generally expect a complex NLU model with a large number of parameters to achieve better accuracy, complex models tend to have high latency. Hence, we are interested in exploring the trade-offs between accuracy and latency by optimizing a total of 24 hyperparameters so we can pick a model that offers an overall positive user experience by balancing quality and latency. Specifically, we optimize the 99th percentile of latency across repeated measurements and the accuracy on a held-out data set.

Methods

BO is typically most effective on search spaces with less than 10 to 15 dimensions. To scale to the 24-dimensional search space in this work, we leverage recent work on high-dimensional BO [1]. Figure 2 shows that the model proposed by [1], which uses a sparse axis-aligned subspace (SAAS) prior and fully Bayesian inference, is crucial to achieving good model fits and outperforms a standard Gaussian process (GP) model with maximum a posteriori (MAP) inference on both accuracy and latency objective.

Figure 2: We illustrate the leave-one-out cross-validation performance for the accuracy and latency objectives. We observe that the SAAS model fits better than a standard GP using MAP.

To efficiently explore the trade-offs between multiple objectives, we use the parallel noisy expected hypervolume improvement (qNEHVI) acquisition function [2], which enables evaluating many architectures in parallel (we use a batch size of 16 in this work) and naturally handles the observation noise that is present in both latency and accuracy metrics: prediction latency is subject to measurement error and and accuracy is subject to randomness in NN training due to optimizing parameters using stochastic gradient methods.

Results

We compare the optimization performance of BO to Sobol (quasi-random) search. Figure 3 shows the results, where the objectives are normalized with respect to the production model, making the reference point equal to (1, 1). Using 240 evaluations, Sobol was only able to find two configurations that outperformed the reference point. On the other hand, our BO method was able to explore the trade-offs between the objectives and improve latency by more than 25% while at the same time improving model accuracy.

Figure 3: On the left, we see that Sobol (quasi-random) search is an inefficient approach that only finds two configurations that are better than the reference point (1,1). On the right, our BO method is much more sample-efficient and is able to explore the trade-offs between accuracy and latency.

Why it matters:

This new method has unlocked on-device deployment for this natural language understanding model as well as several other models at Facebook. Our method requires zero code changes to the existing training and evaluation workflows, making it easily generalizable to different architecture search use cases. We hope that machine learning researchers, practitioners, and engineers find this method useful in their applications and foundational for future research on NAS.

Read the full paper:

https://arxiv.org/abs/2106.11890

Citations:

[1] Eriksson, David, and Martin Jankowiak. “High-Dimensional Bayesian Optimization with Sparse Axis-Aligned Subspaces.” Conference on Uncertainty in Artificial Intelligence (UAI), 2021.

[2] Daulton, Samuel, Maximilian Balandat, and Eytan Bakshy. “Parallel Bayesian Optimization of Multiple Noisy Objectives with Expected Hypervolume Improvement.” arXiv preprint arXiv:2105.08195, 2021.

Try it yourself:

Check out our tutorial in Ax showing how to use the open-source implementation of integrated qNEHVI with GPs with SAAS priors to optimize two synthetic objectives.

View tutorial

The post Optimizing model accuracy and latency using Bayesian multi-objective neural architecture search appeared first on Facebook Research.

Read More