From intern to FTE: Four researchers share their journeys with the Facebook Core Data Science team

Facebook’s Core Data Science (CDS) team is pushing the envelope of what’s possible by exploring and solving novel challenges in science and technology. The CDS internship program was designed to provide researchers with the opportunity to explore career paths in diverse fields of research, such as computer science, statistics, optimization, economics, and social science.

CDS interns are immersed in Facebook culture and see the direct impact their research has on product development, user experience, and the research community as a whole. When transitioning to full-time employees, researchers use passion and curiosity to drive their projects. Here, four researchers share their experiences with the program and what inspired them to transition to their full-time roles.

Staying curious while embracing an exploratory mindset

Taylor Brown is a computational social scientist on the CDS team. She has a PhD in sociology from Duke University, where she specialized in culture and computational methods and networks. She also holds an MSc in evidence-based social intervention from the University of Oxford. Taylor develops methods to improve measurement and mitigation of social inequalities and harmful content on Facebook and Instagram while protecting user privacy.

“As part of my CDS internship at Facebook, I was able to join a groundbreaking collaboration with academic researchers to study the role social media played in the 2020 U.S. presidential election,” says Taylor. “All studies in this collaboration are preregistered, and now, as a full-time team member, I am contributing to their publication. This work will no doubt impact many fields of social science for years to come.”

Taylor’s curiosity has driven her research career and led her to a full-time role at Facebook after two attempts to apply for a CDS internship. “I didn’t make it through the interview process when I first applied for an internship at Facebook,” she says. “The experience taught me a lot about Facebook’s culture. I felt supported all the way through, and I was encouraged to apply again when the next round of positions opened. I’ve now met half a dozen CDS team members who also needed to apply twice to be accepted into the intern program. After having this experience, I always tell people to try again if they don’t get through the interview process the first time around.”

Within CDS, team members have autonomy to focus on projects they feel passionately about. For Taylor, it was easier to transition into a full-time role after two internships. “I recommend completing two internships if you can,” she offers. “I used my first internship to understand Facebook’s culture, research tools, and processes. This set me up for success in my second internship, in which I was able to jump right into project work with the team.”

When deciding whether to stay in academia or move into full-time industry research, it’s important to consider what you want from your career and lifestyle. Taylor found that she appreciated the work/life flexibility Facebook offered and that her internship experience debunked some ideas she had about work outside of academia. “I was afraid industry research would be less stimulating than the free thought of academia, but it’s not,” she says. “There’s no shortage of important questions to answer or complex challenges to solve, and I’m constantly enriched by my coworkers, all of whom are incredibly talented scientists and engineers.”

Discovering new perspectives and keeping an open mind

Yanyi Song earned her PhD from the University of Michigan in Ann Arbor and focuses on Bayesian methods, causal inference, and statistical learning. Yanyi transitioned into a full-time research scientist position last year after completing a data science internship. She saw an online post about the internship program and felt the job was an ideal match for her skills, interests, and career aspirations.

During the program, Yanyi worked on optimizing the user experience with Facebook Ads Manager. “I was motivated by the challenge to think from the user’s perspective, which was new for me,” Yanyi says. “We were tasked with building a data-driven model to improve the user experience, and it took a lot of cross-functional collaboration to reach our goals. As an intern, you have access to exceptional researchers who are incredibly supportive. They’re there to help and answer questions and empower you to share your ideas.”

“I also learned how important communication is when working with different departments,” she continues. “CDS works with teams across Facebook, and it’s a much different environment and pace than academia. I recommend anyone interested in joining CDS prepare themselves to ask questions and maintain a growth mindset. At CDS, you get to work on projects you’re passionate about, and there are endless opportunities to learn and try new things.”

Developing new skills and solving complex challenges

Daniel Thomas recently completed a CDS internship and is now a research scientist at Facebook. He holds a PhD in political science from Columbia University, where he studied the dissolution, formation, and reformation of social networks during and after conflict.

“When interning with the CDS team, there are set parameters for your work,” Daniel says. “My internship expanded on my areas of interest and challenged the skills I’d developed during my PhD program. I had the opportunity to work on a project designed to help us evaluate new network disruptions that the Engineering team was implementing on Facebook. I was surprised by the fast-paced environment and how quickly we can make an impact here compared to the research, review, and publishing process in academia. Interns play an important role on teams and are expected to present ideas, recommendations, and share code and early numbers in meetings. Though it was difficult for me to feel OK with bringing unfinished results to these meetings at first, the challenge empowered me to quickly evolve the efficiency of my coding.”

Daniel was inspired by the scalable impact he was making at Facebook and became a full-time member of the CDS team after his internship ended. “The biggest difference I’ve noticed after my transition to a full-time role is the strong sense of ownership you’re given,” he says. “Rather than being assigned specific tasks, you’re responsible for defining your own projects and moving them forward. Facebook gives you a lot of support while you create your research path, which is incredible. I’ve learned to be more proactive and to overcommunicate. Don’t be afraid to ask questions, and set clear expectations to help you manage your time.”

Feeling inspired to make a meaningful impact

A graduate of the University of Minnesota Twin Cities, Saurabh Verma has a PhD and a master’s degree in computer science and is a research scientist on the CDS team. His research focus is in machine learning and covers deep learning models, natural language processing for Q&A systems and machine reading comprehension models, graph neural networks for graph representation learning, knowledge graphs for information retrieval, and social network analysis.

Saurabh wasn’t looking for an internship when he found a passionate team of problem solvers from diverse backgrounds at CDS. “I was presenting a paper at a conference a few years ago and was invited to a Facebook networking event, where I met my future manager,” he says. “I found the interview process really inspiring. My conversations with recruiters and managers revealed vast opportunities to solve novel challenges, including my work on integrity management and abusive account detection during my two internships.”

Researchers can continue in the CDS intern program if they are still working on their PhD when they finish their first three-month internship. “I received a full-time offer after I graduated and completed my second internship last year,” Saurabh says. “As an intern, I was hyperfocused on specific sections of a big project. I’ve enjoyed being challenged to see the broader project vision, from start to finish, in my full-time role.”

He continues, “My advice to anyone considering a CDS internship — and ultimately a full-time research position at Facebook — is to think hard about the work you see yourself doing. Ask yourself: ‘What projects keep me inspired?’ You never know when opportunities will present themselves, but knowing which projects you’re passionate about may help you stumble into something great.”

Are you interested in applying for an internship with the CDS team? Check out the careers page.

The post From intern to FTE: Four researchers share their journeys with the Facebook Core Data Science team appeared first on Facebook Research.

Read More

Announcing the winners of the 2021 Next-generation Data Infrastructure request for proposals

In April 2021, Facebook launched the Next-generation Data Infrastructure request for proposals (RFP). Today, we’re announcing the winners of this award.
VIEW RFPThe Facebook Core Data and Data Infra teams were interested in proposals that sought out innovative solutions to the challenges that still remain in the data management community. Areas of interest included, but were not limited to, the following topics:

  • Large-scale query processing
  • Physical layout and IO optimizations
  • Data management and processing at a global scale
  • Converged architectures for data wrangling, machine learning, and analytics
  • Advances in testing and verification for storage and processing systems

Read our Q&A with database researchers Stavros Harizopoulos and Shrikanth Shankar to learn more about database research at Facebook, the goal of this RFP, and the inspiration behind the RFP.

The team reviewed 109 high-quality proposals, and we are pleased to announce the 10 winning proposals and six finalists. Thank you to everyone who took the time to submit a proposal, and congratulations to the winners.

Research award recipients

Holistic optimization for parallel query processing
Paraschos Koutris (University of Wisconsin–Madison)

SCALER – SCalAbLe vEctor pRocessing of SPJG-Queries
Wolfgang Lehner, Dirk Habich (Technische Universität Dresden)

AnyScale transactions in the cloud
Natacha Crooks, Joe Hellerstein (University of California, Berkeley)

Proudi: Predictability on unpredictable data infrastructure
Haryadi S. Gunawi (University of Chicago)

Making irregular partitioning practical
Spyros Blanas (The Ohio State University)

Dynamic join processing pushdown in Presto
Daniel Abadi, Chujun Song (University of Maryland, College Park)

A learned persistent key-value store
Tim Kraska (Massachusetts Institute of Technology)

Building global-scale systems using a flexible consensus substrate
Faisal Nawab (University of California, Irvine)

Runtime-optimized analytics using compilation hints
Anastasia Ailamaki (Swiss Federal Institute of Technology Lausanne)

Flexible scheduling for machine learning data processing close to storage
Ana Klimovic, Damien Aymon (ETH Zurich)

Finalists

Next generation data provenance/data governance
Tim Kraska, Michael Cafarella, Michael Stonebraker (Massachusetts Institute of Technology)

Optimizing commitment latency for geo-distributed transactions
Xiangyao Yu (University of Wisconsin–Madison)

Semantic optimization of recursive queries
Dan Suciu (University of Washington)

Towards a disaggregated database for future data centers
Jianguo Wang (Purdue University)

Unified data systems for structured and unstructured data
Matei Zaharia, Christos Kozyrakis (Stanford University)

Unifying machine learning and analytics under a single data engine
Stratos Idreos (Harvard University)

The post Announcing the winners of the 2021 Next-generation Data Infrastructure request for proposals appeared first on Facebook Research.

Read More

Announcing the winners of the 2021 Statistics for Improving Insights, Models, and Decisions request for proposals

In April 2021, Facebook launched the 2021 Statistics for Improving Insights, Models, and Decisions request for proposals live at The Web Conference. Today, we’re announcing the winners of this award.
VIEW RFPAt Facebook, our research teams strive to improve decision-making for a business that touches the lives of billions of people across the globe. Making advances in data science methodologies helps us make the best decisions for our community, products, and infrastructure.

This RFP is a continuation of the 2019 and 2020 RFPs in applied statistics. Through this series of RFPs, the Facebook Core Data Science team, Infrastructure Data Science team, and Statistics and Privacy team aim to foster further innovation and deepen their collaboration with academia in applied statistics, in areas including, but not limited to, the following:

  • Learning and evaluation under uncertainty
  • Statistical models of complex social processes
  • Causal inference with observational data
  • Algorithmic auditing
  • Performance regression detection and attribution
  • Forecasting for aggregated time series
  • Privacy-aware statistics for noisy, distributed data sets

The team reviewed 134 high-quality proposals and are pleased to announce the 10 winning proposals below, as well as the 15 finalists. Thank you to everyone who took the time to submit a proposal, and congratulations to the winners.

Research award winners

Breaking the accuracy-privacy-communication trilemma in federated analytics
Ayfer Ozgur (Stanford University)

Certifiably private, robust, and explainable federated learning
Bo Li, Han Zhao (University of Illinois Urbana-Champaign)

Experimental design in market equilibrium
Stefan Wager, Evan Munro, Kuang Xu (Stanford University)

Learning to trust graph neural networks
Claire Donnat (University of Chicago)

Negative-unlabeled learning for online datacenter straggler prediction
Michael Carbin, Henry Hoffmann, Yi Ding (Massachusetts Institute of Technology)

Non-parametric methods for calibrated hierarchical time-series forecasting
B. Aditya Prakash, Chao Zhang (Georgia Institute of Technology)

Privacy in personalized federated learning and analytics
Suhas Diggavi (University of California Los Angeles)

Reducing simulation-to-reality gap as remedy to learning under uncertainty
Mahsa Baktashmotlagh (University of Queensland)

Reducing the theory-practice gap in private and distributed learning
Ambuj Tewari (University of Michigan)

Robust wait-for graph inference for performance diagnosis
Ryan Huang (Johns Hopkins University)

Finalists

An integrated framework for learning and optimization over networks
Eric Balkanski, Adam Elmachtoub (Columbia University)

Auditing bias in large-scale language models
Soroush Vosoughi (Dartmouth College)

Cross-functional experiment prioritization with decision maker in-the-loop
Emma McCoy, Bryan Liu (Imperial College London)

Data acquisition and social network intervention codesign: Privacy and equity
Amin Rahimian (University of Pittsburgh)

Efficient and practical A/B testing for multiple nonstationary experiments
Nicolò Cesa-Bianchi, Nicola Gatti (Università degli Studi di Milano)

Empirical Bayes deep neural networks for predictive uncertainty
Xiao Wang, Yijia Liu (Purdue University)

Global forecasting framework for large scale hierarchical time series
Rob Hyndman, Christoph Bergmeir, Kasun Bandara, Shanika Wickramasuriya (Monash University)

High-dimensional treatments in causal inference
Kosuke Imai (Harvard University)

Nowcasting time series aggregates: Textual machine learning analysis
Eric Ghysels (University of North Carolina at Chapel Hill)

Online sparse deep learning for large-scale dynamic systems
Faming Liang, Dennis KJ Lin, Qifan Song (Purdue University)

Optimal use of data for reliable off-policy policy evaluation
Hongseok Namkoong (Columbia University)

Principled uncertainty quantification for deep neural networks
Tengyu Ma, Ananya Kumar, Jeff Haochen (Stanford University)

Reliable causal inference with continual learning
Sheng Li (University of Georgia)

Training individual-level machine learning models on noisy aggregated data
Martine De Cock, Steven Golob (University of Washington Tacoma)

Understanding instance-dependent label noise: Learnability and solutions
Yang Liu (University of California Santa Cruz)

The post Announcing the winners of the 2021 Statistics for Improving Insights, Models, and Decisions request for proposals appeared first on Facebook Research.

Read More

What it’s like being a senior researcher at Facebook Core Data Science

Facebook’s Core Data Science (CDS) team is pushing the envelope of what’s possible by exploring and solving novel challenges in science and technology. Senior researchers bring experience across disciplines that range from computer science and statistics to political science and sociology, using data to drive decision-making across product teams. “Our data supports product development across Facebook, and we see our results come to life as optimization, new features, and programs,” explains Shawndra Hill, CDS Research Scientist and Manager.

The unique cross-functional nature of the CDS team and available growth opportunities for senior researchers help set Facebook apart from academic research and other technology companies. Three team members share more about their areas of focus, what a day on the CDS team is like, and their best advice for helping senior researchers succeed.

Staying curious while using data to drive decisions

Ahmed Medhat is a post-graduate of Oxford University, where he studied collaborative behavior in online networks. He joined Facebook five years ago as a staff data scientist before transitioning into a research scientist role with the CDS team.

“Working as part of Facebook Data for Good, my current focus is on creating privacy-safe data sets and tools for addressing some of the world’s greatest humanitarian issues,” Ahmed explains. “Working on this as a research scientist, I’m challenged to find ways to help communities across the world that are both complementary to other resources provided by humanitarian organizations and as scalable and uniform as possible across different geographical regions. While with research in academia it can take some time to sense the real-world impact of one’s research, at Facebook we have a unique opportunity to see the direct impact our work has on the wider research community more quickly. For example, we created tools that can help organizations respond to the Covid-19 pandemic, such as looking at how populations are responding to physical distancing measures, to inform researchers and public health experts.”

Ahmed and his teams are driven by the opportunity to drive life-changing projects and offer something new. “Our work calls for a lot of autonomy and self-accountability,” he shares. “Staying curious and bringing a fresh perspective is important. For those interested in joining the CDS team as a senior researcher, my advice is to demonstrate the unique skills and perspectives that you can bring to Facebook. We also love to see adaptability and the ability to wear different hats. At this level, you’re expected to see beyond the daily minutiae of analysis and code writing, towards work that broadly impacts the company’s product direction and pushes the state of the art in your research area.”

Using passion to drive your research and career path

The CDS team’s Shawndra Hill is also a part-time senior marketing lecturer at Columbia University. She has a PhD in management information systems from NYU Stern School of Business, and prior to her role at Facebook, she was a Senior Principal Researcher at Microsoft Research and a professor at the Wharton School of the University of Pennsylvania. As a manager, Shawndra oversees projects while empowering her team to grow and succeed. Her current research is focused on deriving value from social networks and online behaviors for a range of Facebook’s business applications — specifically for advertising applications that bring people closer to the things they want and need.

“At the senior level, having strong project management skills is critical,” says Shawndra, in reference to her ability to split her time between leadership and individual project work. “During the interview process with Facebook, I saw limitless growth opportunities within the company because of the scale of data science problems, and was also specific about my desire to grow into a management position. With the support and opportunities available within Facebook Research, I was able to reach my goals of leading Facebook scale advertising projects as an individual contributor and becoming a manager within a year.”

Now, Shawndra says, leading a passionate team of researchers is one of her favorite parts of her role. “We’re collaborating in a fast-paced environment. People at Facebook want to translate research into product impact in a short amount of time, and for a researcher, that’s an exciting opportunity that doesn’t often exist in other career paths like academia. To succeed in this field, I suggest identifying the areas you’re most interested in and using that knowledge to find projects at the intersection of your passion and company priorities. From there, you’ll be well positioned to develop a relevant specialty for Facebook and prioritize projects for company impact. As a senior researcher, your experience and deep understanding of relevant research methodologies will add value to teams while also helping them to balance rigor with getting things done. My advice for other senior scientists is to be consistently excellent, someone others want to go to, the person they know they can depend on for quality output.”

Exploring every opportunity to find an area of focus

Ami Tavory holds a PhD in information theory from Tel Aviv University and is a Research Scientist on the CDS team. He was struck by Facebook’s collaborative structure upon joining nearly four years ago, and he found his place working closely with several highly skilled product teams, specializing in detecting and preventing fraud with machine learning in F2 (Facebook Financial Services).

“I’ve been continually impressed with how intentional the Research organization is about building and connecting groups,” he shares. “I’ve worked with several product teams, and have been surprised by the level of collaboration and support. This allows me to find several areas and projects that are the best fit for my interests and my experience. ”

Ami highlights the autonomy at Facebook as another unique aspect of being a part of the team, and says that for a senior researcher with deep experience, it’s an empowering benefit. “Take control of your personal path by exploring every opportunity available to you at Facebook,” he explains. “It’s important to find a combination of something you enjoy that also brings value to the team. Once you find this balance, things will naturally fall into place. We have people on our team who have successfully switched between a management track and a more technical path, and vice versa.

I’ve been blown away by the level of people with whom I work; on some occasions, I’ve read published studies only to realize that the authors are actually part of the Facebook team and happy to collaborate. No two days are the same here, and you’ll find endless opportunities to collaborate on solving complex challenges at scale.”

Interested in learning more about the CDS team? Check out their research team page.

The post What it’s like being a senior researcher at Facebook Core Data Science appeared first on Facebook Research.

Read More

Testing product changes with network effects

This project is collaborative work among the Facebook Core Data Science team, the Experimentation Platform team, and the Messenger team.

What the research is:

Experimentation is ubiquitous in online services such as Facebook, where the effects of product changes are explicitly tested and analyzed in randomized trials. Interference, sometimes referred to as network effects in the context of online social networks, is a threat to the validity of these randomized trials as the presence of interference violates the stable unit treatment value assumption (SUTVA) important to the analysis of these experiments. Colloquially, interference means that an experimental unit’s response to an intervention depends not just on its own treatment, but also on other units’ treatments. For example, consider a food delivery marketplace that tests a treatment that causes users to order deliveries faster. This could reduce the supply of delivery drivers to users in the control group, leading the experimenter to overstate the effects of the treatment.


Figure 1. An illustrative cartoon showing potential interference between test and control units and how cluster randomization accounts for the within-cluster interference.

In our paper we propose a network experimentation framework, which accounts for partial interference between experimental units through cluster randomization (Fig. 1). The framework has been deployed at Facebook at scale, is as easy to use as other conventional A/B tests at Facebook, and has been used by many product teams to measure the effects of product changes. On the design side, we find imbalanced clusters are often superior in terms of bias-variance trade-off than balanced clusters often used in past research. On the analysis side, we introduce a cluster-based regression adjustment that substantially improves precision for estimating treatment effects as well as testing for interference as part of our estimation procedure. In addition, we show how logging which units receive treatment, so-called trigger logging, can be leveraged for even more variance reduction.

While interference is a widely acknowledged issue with online field experiments, there is less evidence from real-world experiments demonstrating interference in online settings. By running many network experiments, we have found a number of experiments with apparent and substantive SUTVA violations. In our paper, two experiments, a Stories experiment using social graph clustering and a Commuting Zones experiment based on geographic clustering, are described in detail, showing significant network effects and demonstrating the value of this experimentation framework.

How it works:

Network experiment design

The design of network experimentation has two primary components: treatment assignment and clustering of experimental units. The component that deploys treatments is depicted visually in Figure 2, where the figure should be read from left to right. A clustering of experimental units, represented by larger circles encompassing colored dots for units, is taken as input. A given clustering and the associated units are considered as a universe, the population under consideration. These clusters of experimental units are deterministically hashed into universe segments based on the universe name, which are then allocated to experiments. Universe segments allow a universe to contain multiple mutually exclusive experiments at any given time, a requirement for a production system used by engineering teams. After allocation to an experiment, segments are randomly split via a deterministic hash based on the experiment name into unit-randomized segments and/or cluster-randomized segments. The final condition allocation deterministically hashes units or clusters into treatment conditions, depending on whether the segment has been allocated to unit or cluster randomization. The result of this final hash produces the treatment vector that is used for the experiment.


Figure 2. Visualization of the network experiment randomization process.

The other main component of network experimentation is clustering of experimental units. An ideal clustering will include all interference within clusters so that there is no interference between clusters, which removes the bias in our estimators. A naive approach that captures all interference is grouping all units into a giant single cluster. This is unacceptable, though, since a cluster-randomized experiment should also have enough statistical power to detect treatment effects. A single cluster including all units has no power, and a clustering that puts every unit in its own cluster, equivalent to unit randomization, leads to good power but captures no interference. This is essentially a bias-variance trade-off: More captured interference leads to less bias, while more statistical power requires smaller clusters. In our paper, we consider two prototypical clustering algorithms due to their scalable implementation: Louvain community detection and recursive balanced partitioning. We find that imbalanced graph clusters generated by Louvain are typically superior in terms of the bias-variance trade-off for graph-cluster randomization.

Network experiment analysis

We are mainly interested in the average treatment effect (ATE) of an intervention (a product change or a new feature), the average effect when the intervention is applied to all users. Many estimation methods exist for ATE for cluster-randomized trials, from methods via cluster-level summaries, to mixed effect models, to generalized estimating equations. For the purpose of easy implementation at scale and explainability, the difference-in-means estimator, i.e., test_mean – control_mean, is used in our framework. The details of the estimands and estimators can be found in our paper. Here we briefly present our two methodological innovations for variance reduction: agnostic regression adjustment and trigger logging (logging units that receive the intervention). Variance reduction is essential since cluster-randomized experiments typically have less power than unit-randomized ones. In our framework, we use the contrast across conditions of pretreatment metrics as covariates to perform regression adjustment. We show that the adjusted estimator is asymptotically unbiased with a much smaller variance. Additionally, trigger logging allows us to perform estimation of the ATE using only the units actually exposed in the experiment. Under mild assumptions, we show that the ATE on the exposed units is equivalent to the ATE on all units that are assigned to the experiment. In Fig. 3, it is shown, for seven metrics in a Stories experiment, how point estimates and CI’s change if we perform an Intent-to-Treat (ITT) analysis on the triggered clusters, instead of triggered users, and if we do not use regression adjustment. The variance reduction from regression adjustment and trigger logging is significant.


Figure 3. Comparison of ATE estimates with scaled 95 percent confidence intervals computed on triggered users and triggered clusters (ITT), with and without regression adjustment (RA) for cluster test and control in a Stories experiment.

Use case: Commuting Zones experiment

We describe in this blog a Commuting Zones experiment as an illustrative example. Commuting Zones, as shown in Fig. 4, are a Facebook Data for Good product and can be used as a geographic clustering for network experiments at Facebook. For products like Jobs on Facebook (JoF), geographical clusters may be especially appropriate as individuals are likely to interact with employers closer to their own physical location. To demonstrate the value of network experimentation, we conducted a mixed experiment, running side-by-side unit-randomized and cluster-randomized experiments, for a JoF product change that up-ranks jobs with few previous applications.


Figure 4. Facebook Commuting Zones in North America


Table 1. Commuting Zone experiment results

Table 1 summarizes the results of this experiment. In the user-randomized test, applications to jobs with no previous applications increased by 71.8 percent. The cluster-randomized conditions, however, showed that these estimates were upwardly biased, and we saw a 49.7 percent increase instead. This comparison benefited substantially from regression adjustment, which can reduce the confidence interval size in Commuting Zone experiments by over 30 percent.

By randomizing this experiment at the Commuting Zone level, the team also confirmed that changes to the user experience that increase this metric can cause employers to post more jobs on the platform (the probability that an employer posted another job increased 17 percent). Understanding the interactions between applicants and employers in a two-sided marketplace is important for the health of such a marketplace, and through network experiments we can better understand these interactions.

Why it matters:

Experimentation with interference has been researched for many years due to its practical importance across different industries. Our paper introduced a practical framework for designing, implementing, and analyzing network experiments at scale. This framework allows us to better predict what will happen when we launch a product or ship a product change to Facebook apps.

Our implementation of network experimentation accommodates mixed experiments, cluster updates, and the need to support multiple concurrent experiments. The simple analysis procedure we present results in substantial variance reduction by leveraging trigger logging as well as our novel cluster-based regression adjusted estimator. We also introduce a procedure for evaluating clusters, which indicates that bias-variance trade-offs are in favor of imbalanced clusters and allows researchers to evaluate these trade-offs for any clustering method they would like to explore. We hope that experimenters and practitioners find this framework useful in their applications and that insights from the paper will foster future research in design and analysis of experiments under interference.

Read the full paper:

Network experimentation at scale

The post Testing product changes with network effects appeared first on Facebook Research.

Read More

Facebook Fellow Spotlight: Striving for provable guarantees in the theoretical foundations of machine learning

Each year, PhD students from around the world apply for the Facebook Fellowship, a program designed to encourage and support doctoral students engaged in innovative and relevant research in areas related to computer science and engineering.

As a continuation of our Fellowship spotlight series, we’re highlighting 2020 Facebook Fellow in applied statistics Lydia Zakynthinou.

Lydia is a PhD candidate at the Khoury College of Computer Science at Northeastern University, where she is advised by Jonathan Ullman and Huy Lê Nguyễn. Her research focuses on the theoretical foundations of machine learning and data privacy.

During her studies at the National Technical University of Athens in Greece, Lydia developed an interest in the theoretical foundations of machine learning and algorithms. Algorithms in particular fascinated her, as they have a direct application in solving real-world problems, especially in a world that values big data.

“Algorithms are everywhere,” Lydia says. “But there is a challenge in determining the trade-offs between the resources they consume, such as computational speed, accuracy, privacy loss, and amount of data, so that we, as researchers, can make informed choices about the algorithms we use.” She points to a simple example of such a trade-off: “Sometimes training a whole deep neural network is really slow, but it is the best we have in terms of accuracy.” That is what encouraged Lydia to study the theoretical foundations of machine learning more deeply.

Lydia’s research seeks to answer two main questions:

  • How can one ensure that an algorithm generalizes well and doesn’t overfit the data set?
  • How can one ensure that the privacy of the individuals’ data is guaranteed?

The effectiveness of an algorithm hinges upon its ability to learn about the population it applies to. But algorithms are designed to learn and be accurate on the data set they are trained on, which leads to two undesirable phenomena: overfitting (that is, an algorithm, misleadingly, performing extremely well on the data set but not on the population) and privacy leakage. This is where generalization and differential privacy come in, respectively.

If an algorithm generalizes well, then its performance on the data set is guaranteed to be close to its performance on the population. Currently, there are many frameworks that seek to achieve this, but they are often incompatible with one another. Lydia’s work proposes a new framework that unifies current theories aiming to understand the properties that an algorithm needs to have to guarantee generalization.

Differential privacy deals with the second side effect, privacy leakage. It is a mathematically rigorous technique that essentially guarantees that no attacker, regardless of their additional knowledge, can infer much more about any individual than they could have had that individual’s data never been included in the data set. It has become the standard criterion for ensuring privacy in machine learning models and has been adopted in several real-world applications. “By design, differential privacy also ensures generalization,” Lydia stresses.

Lydia’s work analyzes core statistical problems and proposes a theoretical framework that unifies current theories, making it possible to create new algorithms that achieve differential privacy and generalize well to the population they apply to. “In general, we should strive toward provable guarantees,” Lydia says, and especially when it comes to data privacy. “Because machine learning is so applied, I feel the need to make sure [an algorithm] behaves as we think it does.”

To learn more about Lydia Zakynthinou and her research, visit her website.

The post Facebook Fellow Spotlight: Striving for provable guarantees in the theoretical foundations of machine learning appeared first on Facebook Research.

Read More

Q&A with Georgia Tech’s Amy Bruckman, research award recipient in online content governance

In this monthly interview series, we turn the spotlight on members of the academic community and the important research they do — as thought partners, collaborators, and independent contributors.

For August, we nominated Amy Bruckman, a professor at Georgia Tech. Bruckman is a winner of the 2019 Content Governance RFP, which sought proposals that helped expand research and advocacy work in the area of online content governance. In this Q&A, Bruckman shares more about her area of specialization, her winning research proposal, and her upcoming book. She also shares what inspires her in her academic work.

Q: Tell us about your role at Georgia Tech and the type of research you specialize in.

Amy Bruckman: I am professor and senior associate chair in the School of Interactive Computing at Georgia Tech. We are halfway between an I-school and a CS department — more technical than most I-schools and more interdisciplinary than most CS departments.

I founded my first online community in 1993, and I am endlessly fascinated by how the design features of an online environment shape human behavior. My students and I build new tools to support novel kinds of online interaction, and we also study existing systems using a mixed-methods approach. My specialty is qualitative methods. My students and I participate online and take field notes on what we observe (methods inspired by sociology and anthropology), and we also interview people about their experiences (building on clinical interview techniques from psychology). I partner with people who do big data and NLP research, and I’ve found that qualitative and quantitative methods are usually more powerful when used together.

Q: What have you been working on lately?

AB: Personally, lately I’ve been focused on my book Should You Believe Wikipedia? Online Communities and the Construction of Knowledge. It is coming out in January from Cambridge University Press. In the book, I try to explain how online communities are designed, with a particular focus on how people can collaboratively build knowledge.

Q: You were a winner of the 2019 Content Governance RFP. What was your winning proposal about?

AB: Our research asks the question, What happens after a controversial figure who regularly breaks platform rules is kicked off, or “deplatformed”? In particular, we studied what happened after Alex Jones, Milo Yiannopoulos, and Owen Benjamin were kicked off Twitter.

Q: What were the results of this research?

AB: My coauthors (Shagun Jhaver, Christian Boylston, and Diyi Yang) and I found that deplatforming significantly reduced the number of conversations about those individuals. More important, the overall activity and toxicity levels of supporters declined after deplatforming. For example, Milo encouraged his followers to attack actress Leslie Jones. After he was deplatformed, his supporters were better behaved. The full paper will appear at CSCW 2021.

Q: What inspires you in your academic work?

AB: I believe that our field is at a crossroads: The internet needs some redesign to support healthy communities and a working public sphere. The last chapter of my book is focused on how we can help the internet to bring out the best in us all. I try to work toward that goal in my research and in my teaching. Every fall, I teach our required ethics class “Computing, Society, and Professionalism,” and in spring, I teach “Design of Online Communities.” It’s a privilege to teach students about these issues, and the students have impact as they go on to design and build the information systems we all use every day.

Q: Where can people learn more about you and your work?

AB: My book Should You Believe Wikipedia? will be published in early 2022, and there is a sample chapter on my website.

The post Q&A with Georgia Tech’s Amy Bruckman, research award recipient in online content governance appeared first on Facebook Research.

Read More

Registration now open for the 2021 Instagram Workshop on Recommendation Systems at Scale

Instagram invites the academic community and industry peers to join the first Instagram Workshop on Recommendation Systems at Scale, taking place virtually on Thursday, September 23. Those interested may register at the link below.

Register

Production recommendation systems in large social and commerce platforms introduce complex and unprecedented challenges. The goal of this workshop is to bring together leading researchers and practitioners in related fields to share knowledge and explore research collaboration opportunities between academia and industry.

“Every day, we help over a billion people connect with their friends, interests, businesses, and creators they love on Instagram. To ensure that a community of this size finds new and inspiring content, we need to constantly evolve our recommendation systems technology. Collaborating with like-minded industry and academic experts allows us to do just that,” says Aameek Singh, Engineering Director, Instagram.

Confirmed speakers

Below is the list of confirmed speakers as of today.

  • Anoop Deoras (AWS, Amazon)
  • Diane Hu (Etsy)
  • Jonathan J Hunt (Twitter Cortex Applied Research)
  • Chris Wiggins (The New York Times)

As speakers are confirmed, they will be added to the registration page, along with their full bios and topics. View additional event details and register to attend at the link below.

Register

The post Registration now open for the 2021 Instagram Workshop on Recommendation Systems at Scale appeared first on Facebook Research.

Read More

When do recommender systems amplify user preferences? A theoretical framework and mitigation strategies

What the research is:

Recommender systems have come to influence nearly every aspect of human activity on the internet, whether in the news we read, the products we purchase, or the entertainment we consume. The algorithms and models at the heart of these systems rely on learning our preferences through the course of our interactions with them; when we watch a video or like a post on Facebook, we provide hints to the system about our preferences.

This repeated interplay between people and algorithms creates a feedback loop that results in recommendations that are increasingly customized to our tastes. Ideally, these feedback loops ought to be virtuous all the time; the recommender system is able to infer exactly what our preferences are and provides us with recommendations that enhance the quality of our lives.

However, what happens when the system overindexes and amplifies interactions that do not necessarily capture the user’s true preferences? Or if the user’s preferences have drifted toward recommended items that could be considered harmful or detrimental to their long-term well-being? Under what conditions would recommender systems respond to these changes and amplify preferences leading to a higher prevalence of harmful recommendations?

How it works:

In this paper, we provide a theoretical framework to answer these questions. We model the interactions between users and recommender systems and explore how these interactions may lead to potential harmful outcomes. Our main assumption is that users have a slight inclination to reinforce their opinion (or drift), i.e., increase their preference toward recommendations that they seem to correlate well with, and decrease it otherwise. We characterize the temporal evolution of the user’s preferences as a function of the user, the recommender system, and time, and ask whether this function admits a fixed point, i.e., any change in the system’s response to the user’s interactions does not change their preferences. We show that even under a mild drift and absent any external intervention, no such fixed point exists. That is, even a slight preference by a user for recommendations in a given category can lead to increasingly higher concentrations of item recommendations from that category. We refer to this phenomenon as preference amplification.

Recommender system model

We leverage the well-adopted collaborative filtering model of recommendation systems – each (user, item) tuple receives a score based on the likelihood of the user to be interested in the item. These scores are computed using low-dimension matrix factorization. We use a stochastic recommendation model, in which the set of items presented to a user is chosen probabilistically relative to the items’ scores (rather than deterministically sorting by score). The level of stochasticity in the system is determined by a parameter 𝛽; the higher the 𝛽, the lesser the stochasticity and the distribution of scores is heavily concentrated in the top items. Finally, we think of the content available for recommendation to be benign or problematic, and use ɑ to denote the prevalence of the latter, i.e., the percentage of problematic content out of all content.

Our model also includes the temporal interactions between the user and the recommender system, where in each iteration the user is presented with a set of items, and signals to the recommender system their interests. These interests drift slightly based on the recommended items, the actual magnitude of drift being parameterized by the score the item receives.

The figure below illustrates our temporal drift model. The recommender system initially recommends a diverse set of items to the user, who in turn interacts with those items they prefer. The recommender system picks up this signal, and recommends a less diverse set of items (depicted as only green and blue items) that matches the perceived preferences of the user. The user then drifts further towards a very specific set of items (depicted as the items in blue) that the recommender system suggested. This causes the recommender system to only suggest items from that specific class (blue items).

Simulations

In order to study the parameter space in which the system reinforces recommendation scores, we use simulations with both synthetic and real data sets. We show that the system reinforces scores for items based on the user’s initial preferences — items similar to those liked by the user initially will have a higher likelihood of being recommended over time, and conversely, those that the user did not initially favor will have a decreasing probability of recommendation.

In the figure above on the left, we can see the effect of preference amplification. Solid lines (top group of lines) indicate the likable items, whose probability of receiving a positive reaction from the user is above 0.5. The dashed lines (bottom group) indicate the items that have a low positive reaction from the user. As the figure shows, the probability of liking an item increases toward 1 if its score is positive and toward 0 otherwise. For higher values of 𝛽 (the stochasticity of the recommender system), the stochastic recommender system acts as a Top-N recommender and is therefore more likely to present the users with items that they already liked, leading to stronger reinforcement of their preferences. On the right-side plot in the figure above we see another outcome of preference amplification – the probability of the user liking an item from the top 5% of items recommended to them significantly increases over time. This amplification effect is especially evident for high values of 𝛽, where the stochasticity of the system is low, and the recommender system chooses items that are very likely to be preferred by the user.

Mitigations

Finally, we discuss two strategies for mitigating the effects of preference amplification of problematic entities at a) the global level and b) the personal level. In the former, the strategy is to remove these entities globally in order to reduce their overall prevalence, and in the latter, the system targets users and applies interventions aimed at reducing the probability of recommendation of these entities.


In the figure above we characterize simulation effects of a global intervention on problematic content. We plot the probability of recommending an item of problematic content for different initial prevalences (denoted by ɑ). The figure shows that despite the low prevalence of the problematic content, if there is some initial affinity for that type of content, the probability of it being recommended to the user increases over time.

In the paper, we also describe an experiment we conducted using a real-world large-scale video recommender system. In the experiment, we downrank videos considered to include borderline nudity (the platform already filters out videos that violate community standards) for users who have a high level of exposure to them consistently. The results of the experiment show that in addition to reducing exposure of this content in the impacted population, we saw that overall engagement go up by +2%. These results are highly encouraging, as not only we can prevent exposure to problematic content, we also have an overall positive effect on the user experience.

Why it matters:

In this work, we study the interactions between users and recommender systems, and show that for certain user behaviors, their preferences can be amplified by the recommender system. Understanding the long-term impact of ML systems helps us, as practitioners, to build better safeguards and ensure that our models are optimized to serve the best interests of our users.

Read the full paper:

A framework for understanding preference amplification in recommender systems

Learn More:

Watch our presentation at KDD 2021.

The post When do recommender systems amplify user preferences? A theoretical framework and mitigation strategies appeared first on Facebook Research.

Read More