On the Air: Creative Technology Elevates Broadcast Workflows for International Sporting Event with NVIDIA Networking

Talk about a signal boost. Creative Technology is tackling 4K and 8K signals, as well as new broadcast workflows, with the latest NVIDIA networking technologies.

The London-based firm is one of the world’s leading suppliers of audio visual equipment for broadcasting and online events. Part of global production company NEP Group, CT helps produce high-quality virtual and live events by providing advanced technologies and equipment, from large-screen displays to content delivery systems.

Before the COVID-19 pandemic hit, CT was looking to enhance the broadcast experience, bringing audiences and content closer together. Already in the process of switching from a baseband software-defined infrastructure (SDI) architecture to more advanced internet protocol (IP)-based technologies, CT was prepared when the pandemic led to an increased demand in virtual events.

The company decided to invest in KAIROS, Panasonic’s next-generation IT and IP video processing platform. KAIROS is a software-based, open architecture platform that uses CPU and GPU processing to significantly improve broadcast performance.

CT opted for NVIDIA GPUs to power KAIROS, which uses NVIDIA Rivermax IP streaming acceleration to enable direct data transfers to and from the GPU, leading to enhanced flexibility and increased performance for virtual events.

With plans to use KAIROS for the world’s most recognized sporting event this month, CT is using IP enabled by NVIDIA switches and NVIDIA RTX GPUs. This technology allows CT to easily scale up for larger shows and save time in setting up new productions, while transforming broadcast workflows.

Taking Broadcast Beyond the Standard

With LED screens increasing in resolution, it’s now more common for companies to deal with 4K and 8K signals. CT wanted a powerful solution that could keep up, while also providing better scalability and flexibility to enhance workflows.

When CT first started testing KAIROS, they were discussing using the platform to accommodate a 3G-SDI workflow, which supports the move from 1080/50 interlaced video formats (1080i) to 1080/50 progressive video formats (1080p).

In interlaced scanning, the frame is divided into odd and even lines — only half the frame is shown on screen, and the other half appears in 1/60th of a second. The lines switch so quickly that viewers will see the entire frame, but they may also see flickers on screen.

In progressive scans, the entire frame is transmitted simultaneously. All the lines in the frame are shown at once to fill the screen, which reduces flicker. Progressive scans are ideal for digital transmissions and have become the standard for high-definition TV displays.

But CT also needed to ensure its technology could keep up with any future video workflow advances demanded by clients.

The company has its own servers built on NVIDIA RTX GPUs with ConnectX-6 DX cards, and KAIROS delivers high performance by using the power and flexibility of the GPUs. The CT team no longer has to deal with the painful process of converting 4K and 8K signals to SDI. Instead, it can pass the signals to KAIROS, which can distribute video feeds to projectors or screens regardless of the resolution or format.

“Essentially, what KAIROS did was give us a lot more flexibility,” said Sid Lobb, head of Vision and Integrated Networks at Creative Technology. “There is utter flexibility with what we can use and how we allocate the power that the NVIDIA RTX GPUs provide.”

Switching It Up 

Transitioning from SDI to IP allowed CT to use software for driving all the events. With IP, CT can use a switch instead of cables to connect systems.

“Now, it’s more like connecting computers to each other versus directly connecting cameras to a processor,” said Lobb. “We’re able to use a network to connect the entire production signal path. It’s a whole change to broadcast workflows.”

The latest version of KAIROS enables CT to use the network as a matrix switcher, which allows the team to easily switch from one video or audio source to another. For example, in events that take place in a sports arena, there could be up to 100 PCs capturing and producing different content. During the event, CT could be switching from one PC to another, which would’ve been challenging with traditional architectures. But with IP, CT can easily switch among sources, and also scale up and down to different size shows using the same solution.

The team is also experiencing massive time savings when it comes to getting new productions up and running, as the programming of KAIROS is intuitive and efficient. Each virtual event is different, but KAIROS makes it easy for CT to configure input and outputs based on their productions.

The team will use GPU-powered solutions to enhance the experience for future broadcasting and live events.

The post On the Air: Creative Technology Elevates Broadcast Workflows for International Sporting Event with NVIDIA Networking appeared first on The Official NVIDIA Blog.

Read More

NVIDIA-Certified Systems Land on the Desktop

Enterprises challenged with running accelerated workloads have an answer: NVIDIA-Certified Systems. Available from nearly 20 global computer makers, these servers have been validated for running a diverse range of accelerated workloads with optimum performance, reliability and scale.

Now NVIDIA-Certified Systems are expanding to the desktop with workstations that undergo the same testing to validate their ability to run GPU-accelerated applications well.

Certification ensures that these systems, available as desktop or laptop models, have a well-balanced design and the correct configurations to maximize performance. GPUs eligible for certification in the workstations include the newest NVIDIA RTX A6000, A5000 and A4000, as well as the RTX 8000 and 6000.

NVIDIA-Certified workstations will join a lineup of over 90 already available systems that range from the highest performance AI servers with the NVIDIA HGX A100 8-GPU, to enterprise-class servers with the NVIDIA A30 Tensor Core GPU for mainstream accelerated data centers, to low-profile, low-power systems designed for the edge with NVIDIA T4 GPUs.

Certified Systems to Accelerate Data Science on CDP

Cloudera Data Platform (CDP) v7.1.6, which went into general availability last week, now takes advantage of NVIDIA-Certified Systems. This latest version adds RAPIDS to accelerate data analytics, ETL and popular data science tools like Apache Spark with NVIDIA GPUs to churn through massive data operations.

Testing has shown that this version of CDP runs up to 10x faster on servers with NVIDIA GPUs vs. non-accelerated servers. To make it easy to get started, NVIDIA and Cloudera recommend two NVIDIA-Certified server configurations that customers can purchase from several vendors:

  • CDP-Ready: For running Apache Spark, a CDP-Ready configuration of NVIDIA-Certified servers with two NVIDIA A30 GPUs per server offers over 5x the performance at less than 50 percent incremental cost relative to modern CPU-only alternatives.
  • AI ready: For customers additionally running machine learning or other AI-related applications, the NVIDIA A100 GPU provides even more performance — as well as acceleration on machine learning and AI training.

Data scientists often develop and refine machine learning and deep learning models on workstations to augment data center resources or help minimize cloud-based compute costs. By using an NVIDIA-Certified workstation, they can transition their work to NVIDIA-Certified servers when it’s time for larger scale prototyping and eventually production, without having to port to a different tool or framework.

New White Paper Describes Value of Certification

When it comes to installing GPUs and SmartNICs in a system, choosing the right server or workstation model and correctly configuring the components and firmware are critical to getting the most out of the investment.

With NVIDIA-Certified Systems, NVIDIA and its partners have already done the work of validating that a particular system is capable of running accelerated workloads well, and they’ve figured out the most optimal hardware configuration.

Misconfiguration can lead to poor performance and even inability to function properly or complete tasks. The certification process ensures that issues such as these are surfaced and resolved for each tested system. We’ve described this and more in a new white paper, Accelerate Compute-Intensive Workloads with NVIDIA-Certified Systems.

Our system partners run a suite of more than 25 tests designed by NVIDIA based on our vast experience with compute, graphics and network acceleration. Each of the tests is chosen to exercise the hardware of the system in a unique and thorough manner, so as many potential configuration issues as possible can be exposed. Some of the tests focus on a single aspect of the hardware, while others stress multiple components, both simultaneously as well as in a multi-step workflow.

With NVIDIA-Certified Systems, enterprises can confidently choose performance-optimized hardware to power their accelerated computing workloads — from the desktop to the data center to the edge.

Learn more about NVIDIA-Certified Systems:

The post NVIDIA-Certified Systems Land on the Desktop appeared first on The Official NVIDIA Blog.

Read More

Leading Lights: NVIDIA Researchers Showcase Groundbreaking Advancements for Real-Time Graphics

Computer graphics and AI are cornerstones of NVIDIA. Combined, they’re bringing creators closer to the goal of cinema-quality 3D imagery rendered in real time.

At a series of graphics conferences this summer, NVIDIA Research is sharing groundbreaking work in real-time path tracing and content creation, much of it based on cutting-edge AI techniques. These projects are tackling the hardest unsolved problems in graphics with new tools that advance the state of the art in real-time rendering.

One goal is improving the realism of rendered light as it passes through complex materials like fur or fog. Another is helping artists more easily turn their creative visions into lifelike models and scenes.

Presented at this week’s SIGGRAPH 2021 — as well as the recent High-Performance Graphics conference and the Eurographics Symposium on Rendering — these research advancements highlight how NVIDIA RTX GPUs make it possible to further the frontiers of photorealistic real-time graphics.

Rendering photorealistic images in real time requires accurate simulation of light, modeling the same laws that govern light in the physical world. The most effective approach known so far, path tracing, requires massive computational resources but can deliver spectacular imagery.

The NVIDIA RTX platform, with dedicated ray-tracing hardware and high-performance Tensor Cores for efficient evaluation of AI models, is tailor made for this task. Yet there are still situations where creating high-fidelity rendered images remains challenging.

Consider, for one, a tiger prowling through the woods.

Seeing the Light: Real-Time Path Tracing

To make a scene completely realistic, creators must render complex lighting effects such as reflections, shadows and visible haze.

In a forest scene, dappled sunlight filters through the leaves on the trees and grows hazy among the water molecules suspended in the foggy air. Rendering realistic real-time imagery of clouds, dusty surfaces or mist like this was once out of reach. But NVIDIA researchers have developed techniques that often compute the visual effect of these phenomena 10x more efficiently.

The tiger itself is both illuminated by sunlight and shadowed by trees. As it strides through the woods, its reflection is visible in the pond below. Lighting these kinds of rich visuals with both direct and indirect reflections can require calculating thousands of paths for every pixel in the scene.

It’s a task far too resource-hungry to solve in real time. So our research team created a path-sampling algorithm that prioritizes the light paths and reflections most likely to contribute to the final image, rendering images over 100x more quickly than before.

AI of the Tiger: Neural Radiance Caching

Another group of NVIDIA researchers achieved a breakthrough in global illumination with a new technique named neural radiance caching. This method uses both NVIDIA RT Cores for ray tracing and Tensor Cores for AI acceleration to train a tiny neural network live while rendering a dynamic scene.

The neural network learns how light is distributed throughout the scene. It evaluates over a billion global illumination queries per second when running on an NVIDIA GeForce RTX 3090 GPU, depicting the tiger’s dense fur with rich lighting detail previously unattainable at interactive frame rates.

Seamless Creation of Tough Textures

As rendering algorithms have progressed, it’s crucial that the 3D content available keeps up with the complexity and richness that the algorithms are capable of.

NVIDIA researchers are diving into this area by developing a variety of techniques that support content creators in their efforts to model rich and realistic 3D environments. One area of focus is on materials with rich geometric complexity, which can be difficult to simulate using traditional techniques.

The weave of a polo shirt, the texture of a carpet, or blades of grass have features often much smaller than the size of a pixel, making it difficult to efficiently store and render representations of them. NVIDIA researchers are addressing this with NeRF-Tex, an approach that uses neural networks to represent these challenging materials and encode how they respond to lighting.

Seeing the Forest for the Trees

Complex geometric objects also vary in their appearance depending on how close they are to the viewer. A leafy tree is one example: Close up, there’s enormous detail in its branches, leaves and bark. From afar, it may appear to be little more than a green blob.

It would be a waste of time to render detailed bark and leaves on a tree that’s on the other end of the forest in a scene. But when zooming in for a close-up, the model should be as realistic as possible.

This is a classic problem in computer graphics known as level of detail. Artists have often been burdened with this challenge, manually modeling multiple versions of each 3D object to enable efficient rendering.

NVIDIA researchers have developed a new approach that generates simplified models automatically based on an inverse rendering method. With it, creators can generate simplified models that are optimized to appear indistinguishable from the originals, but with drastic reductions in their geometric complexity.

NVIDIA at SIGGRAPH 2021 

More than 200 scientists around the globe make up the NVIDIA Research team, focusing on AI, computer graphics, computer vision, self-driving cars, robotics and more. At SIGGRAPH, which runs from Aug. 9-13, our researchers are presenting the following papers:

Don’t miss NVIDIA’s special address at SIGGRAPH on Aug. 10 at 8 a.m. Pacific, revealing our latest technology, demos and more. Catch our Real Time Live demo on Aug. 10 at 4:30 p.m. Pacific to see how NVIDIA Research creates AI-driven digital avatars.

We’re also discussing esports as a real-time graphics challenge in a panel on Aug. 11. An interactive esports demo is available on demand through the SIGGRAPH Emerging Technologies program.

For more, check out the full lineup of NVIDIA events at SIGGRAPH 2021.

The post Leading Lights: NVIDIA Researchers Showcase Groundbreaking Advancements for Real-Time Graphics appeared first on The Official NVIDIA Blog.

Read More

Time to Embark: Autonomous Trucking Startup Develops Universal Platform on NVIDIA DRIVE

Autonomous trucking startup Embark is planning for universal autonomy of commercial semi-trucks, developing one AI platform that fits all.

The company announced today that it will use NVIDIA DRIVE to develop its Embark Universal Interface (EUI), a manufacturer-agnostic platform that includes the compute and multimodal sensors necessary for autonomous trucks. This flexible approach, combined with the high performance of NVIDIA DRIVE, leads to an easily scalable solution for safer, more efficient delivery and logistics.

The EUI is purpose-built to run Embark Driver autonomous driving software for a comprehensive self-driving trucking system.

Most trucking carriers don’t just use one model of vehicle in their fleets. This variety can even extend to vehicles from different manufacturers to haul a wide range of cargo around the world.

The Embark platform will be capable of integrating into trucks from any of the four major truck manufacturers in the U.S. — PACCAR, Volvo, International and Freightliner. By developing a platform that can be retrofitted to such a wide range of vehicles, Embark is helping the trucking industry realize the benefits of AI-powered driving without having to wait for purpose-built vehicles.

And with NVIDIA DRIVE at its core, the platform leverages the best in high-performance AI compute for robust self-driving capabilities.

Scaling Safety

Autonomous vehicles are always learning, taking in vast amounts of data to navigate the unpredictability of the real world, from highways to crowded ports. This rapid processing requires centralized, high-performance AI compute.

The NVIDIA DRIVE platform is the first scalable AI hardware and software platform to enable the production of automated and self-driving vehicles. It combines deep learning, sensor fusion and surround vision for a safe driving experience.

This end-to-end open platform allows for one development investment across an entire fleet, from level 2+ systems all the way to level 5 fully autonomous vehicles. In addition to high-performance, scalable compute, the EUI will have all the necessary functional safety certification to operate without a driver on public roads.

“We need an enormous amount of compute horsepower in our trucks,” said Ajith Dasari, head of Hardware Platform at Embark. “NVIDIA DRIVE meets this need head-on, and allows us to outfit our partners and customers with the best self-driving hardware and software currently on the market.”

A Growing Ecosystem

Embark is already working with leading trucking companies and plans to continue to extend its software and hardware technology.

In April, the company unveiled partnerships with Werner Enterprises, Mesilla Valley Transportation and Bison Transport. It’s also working with shippers including Anheuser Busch InBev and HP, Inc.

Embark plans to list on the public market, announcing a SPAC, or special purpose acquisition company, agreement in June, as well as a partnership with Knight-Swift Transportation. The autonomous trucking company will join the ranks of NVIDIA DRIVE ecosystem members who have collectively raised more than $8 billion via public listings.

And just like the trucks running on its Embark Universal Interface, the company is tapping the power of NVIDIA DRIVE to keep traveling further and more intelligently.

The post Time to Embark: Autonomous Trucking Startup Develops Universal Platform on NVIDIA DRIVE appeared first on The Official NVIDIA Blog.

Read More

What Matters in Learning from Offline Human Demonstrations for Robot Manipulation

Overview

Imitation Learning is a promising approach to endow robots with various complex manipulation capabilities. By allowing robots to learn from datasets collected by humans, robots can learn to perform the same skills that were demonstrated by the human. Typically, these datasets are collected by having humans control robot arms, guiding them through different tasks. While this paradigm has proved effective, a lack of open-source human datasets and reproducible learning methods make assessing the state of the field difficult. In this paper, we conduct an extensive study of six offline learning algorithms for robot manipulation on five simulated and three real-world multi-stage manipulation tasks of varying complexity, and with datasets of varying quality. Our study analyzes the most critical challenges when learning from offline human data for manipulation.

Based on the study, we derive several lessons to understand the challenges in learning from human demonstrations, including the sensitivity to different algorithmic design choices, the dependence on the quality of the demonstrations, and the variability based on the stopping criteria due to the different objectives in training and evaluation. We also highlight opportunities for learning from human datasets, such as the ability to learn proficient policies on challenging, multi-stage tasks beyond the scope of current reinforcement learning methods, and the ability to easily scale to natural, real-world manipulation scenarios where only raw sensory signals are available.

We have open-sourced our datasets and all algorithm implementations to facilitate future research and fair comparisons in learning from human demonstration data. Please see the robomimic website for more information.

In this study, we investigate several challenges of offline learning from human datasets and extract lessons to guide future work.

Why is learning from human-labeled datasets difficult?

We explore five challenges in learning from human-labeled datasets.

  • (C1) Unobserved Factors in Human Decision Making. Humans are not perfect Markovian agents. In addition to what they currently see, their actions may be influenced by other external factors – such as the device they are using to control the robot and the history of the actions that they have provided.
  • (C2) Mixed Demonstration Quality. Collecting data from multiple humans can result in mixed quality data, since some people might be better quality supervisors than others.
  • (C3) Dependence on dataset size. When a robot learns from an offline dataset, it needs to understand how it should act (action) in every scenario that it might encounter (state). This is why the coverage of states and actions in the dataset matters. Larger datasets are likely to contain more situations, and are therefore likely to train better robots.
  • (C4) Train Objective ≠ Eval Objective. Unlike traditional supervised learning, where validation loss is a strong indicator of how good a model is, policies are usually trained with surrogate losses. Consider an example where we train a policy via Behavioral Cloning from a set of demonstrations on a block lifting task. Here, the policy is trained to replicate the actions taken by the demonstrator, but this is not necessarily equivalent to optimizing the block lifting success rate (see the Dagger paper for a more precise explanation). This makes it hard to know which trained policy checkpoints are good without trying out each and every model directly on the robot – a time consuming process.
  • (C5) Sensitivity to Agent Design Decisions. Performance can be very sensitive to important agent design decisions, like the observation space and hyperparameters used for learning.

Study Design

In this section, we summarize the tasks (5 simulated and 3 real), datasets (3 different variants), algorithms (6 offline methods, including 3 imitation and 3 batch reinforcement), and observation spaces (2 main variants) that we explored in our study.

Tasks

Lift
Can
Tool Hang
Square
Lift (Real)
Can (Real)
Tool Hang (Real)
Transport
We collect datasets across 6 operators of varying proficiency and evaluate offline policy learning methods on 8 challenging manipulation tasks that test a wide range of manipulation capabilities including pick-and-place, multi-arm coordination, and high-precision insertion and assembly.

Task Reset Distributions

When measuring the task success rate of a policy, the policy is evaluated across several trials. At the start of each trial, the initial placement of all objects in the task are randomized from a task reset distribution. The videos below show this distribution for each task. This gives an impression of the range of different scenarios that a trained policy is supposed to be able to handle.

Lift
Can
Tool Hang
Square
Lift (Real)
Can (Real)
Tool Hang (Real)
Transport
We show the task reset distributions for each task, which governs the initial placement of all objects in the scene at the start of each episode. Initial states are sampled from this distribution at both train and evaluation time.

Datasets

We collected 3 kinds of datasets in this study.

Machine-Generated

These datasets consist of rollouts from a series of SAC agent checkpoints trained on Lift and Can, instead of humans. As a result, they contain random, suboptimal, and expert data due to the varied success rates of the agents that generated the data. This kind of mixed quality data is common in offline RL works (e.g. D4RL, RLUnplugged).

Lift (MG)
Can (MG)
Lift and Can Machine-Generated datasets.

Proficient-Human

These datasets consist of 200 demonstrations collected from a single proficient human operator using RoboTurk.

Lift (PH)
Can (PH)
Square (PH)
Transport (PH)
Tool Hang (PH)
Proficient-Human datasets generated by 1 proficient operator (with the exception of Transport, which had 2 proficient operators working together).

Multi-Human

These datasets consist of 300 demonstrations collected from six human operators of varied proficiency using RoboTurk. Each operator falls into one of 3 groups – “Worse”, “Okay”, and “Better” – each group contains two operators. Each operator collected 50 demonstrations per task. As a result, these datasets contain mixed quality human demonstration data. We show videos for a single operator from each group.

Lift (MH) – Worse
Lift (MH) – Okay
Lift (MH) – Better
Multi-Human Lift dataset. The videos show three operators – one that’s “worse” (left), “okay” (middle) and “better” (right).
Can (MH) – Worse
Can (MH) – Okay
Can (MH) – Better
Multi-Human Can dataset. The videos show three operators – one that’s “worse” (left), “okay” (middle) and “better” (right).
Square (MH) – Worse
Square (MH) – Okay
Square (MH) – Better
Multi-Human Square dataset. The videos show three operators – one that’s “worse” (left), “okay” (middle) and “better” (right).
Transport (MH) – Worse-Worse
Transport (MH) – Okay-Okay
Transport (MH) – Better-Better
Transport (MH) – Worse-Okay
Transport (MH) – Worse-Better
Transport (MH) – Okay-Better
Multi-Human Transport dataset. These were collected using pairs of operators with Multi-Arm RoboTurk (each one controlled 1 robot arm). We collected 50 demonstrations per combination of the operator subgroups.

Algorithms

We evaluated 6 different offline learning algorithms in this study, including 3 imitation learning and 3 batch (offline) reinforcement learning algorithms.

We evaluated 6 different offline learning algorithms in this study, including 3 imitation learning and 3 batch (offline) reinforcement learning algorithms.
  • BC: standard Behavioral Cloning, which is direct regression from observations to actions.
  • BC-RNN: Behavioral Cloning with a policy network that’s a recurrent neural network (RNN), which allows modeling temporal correlations in decision-making.
  • HBC: Hierarchical Behavioral Cloning, where a high-level subgoal planner is trained to predict future observations, and a low-level recurrent policy is conditioned on a future observation (subgoal) to predict action sequences (see Mandlekar*, Xu* et al. (2020) and Tung*, Wong* et al. (2021) for more details).
  • BCQ: Batch-Constrained Q-Learning, a batch reinforcement learning method proposed in Fujimoto et al. (2019).
  • CQL: Conservative Q-Learning, a batch reinforcement learning method proposed in Kumar et al. (2020).
  • IRIS: Implicit Reinforcement without Interaction, a batch reinforcement learning method proposed in Mandlekar et al. (2020).

Observation Spaces

We study two different observation spaces in this work – low-dimensional observations and image observations.

We study two different observation spaces in this work.

Image Observations

We provide examples of the image observations used in each task below.

Most tasks have a front view and wrist view camera. The front view matches the view provided to the operator during data collection.
Tool Hang has a side view and wrist view camera. The side view matches the view provided to the operator during data collection.
Transport has a shoulder view and wrist view camera per arm. The shoulder view cameras match the views provided to each operator during data collection.

Summary of Lessons Learned

In this section, we briefly highlight the lessons we learned from our study. See the paper for more thorough results and discussion.

Lesson 1: History-dependent models are extremely effective.

We found that there is a substantial performance gap between BC-RNN and BC, which highlights the benefits of history-dependence. This performance gap is larger for longer-horizon tasks (e.g. ~55% for the Transport (PH) dataset compared to ~5% for the Square (PH) dataset)) and also larger for multi-human data compared to single-human data (e.g.~25% for Square (MH) compared to ~5% for Square (PH)).

Methods that make decisions based on history, such as BC-RNN and HBC, outperform other methods on human datasets.

Lesson 2: Batch (Offline) RL struggles with suboptimal human data.

Recent batch (offline) RL algorithms such as BCQ and CQL have demonstrated excellent results in learning from suboptimal and multi-modal machine-generated datasets. Our results confirm the capacity of such algorithms to work well – BCQ in particular performs strongly on our agent-generated MG datasets that consist of a diverse mixture of good and poor policies (for example, BCQ achieves 91.3% success rate on Lift (MG) compared to BC which achieves 65.3%).

Surprisingly though, neither BCQ nor CQL performs particularly well on these human-generated datasets. For example, BCQ and CQL achieve 62.7% and 22.0% success respectively on the Can (MH) dataset, compared to BC-RNN which achieves 100% success. This puts the ability of such algorithms to learn from more natural dataset distributions into question (instead of those collected via RL exploration or pre-trained agents). There is an opportunity for future work in batch RL to resolve this gap.

While batch (offline) RL methods are proficient at dealing with mixed quality machine-generated data, they struggle to deal with mixed quality human data.
To further evaluate methods in a simpler setting, we collected the Can Paired dataset, where every task instance has two demonstrations, one success and one failure. Even this simple setting, where each start state has exactly one positive and one negative demonstration, poses a problem.

Lesson 3: Improving offline policy selection is important.

The mismatch between train and evaluation objective causes problems for policy selection – unlike supervised learning, the best validation loss does not correspond to the best performing policy. We found that the best validation policy is 50 to 100% worse than the best performing policy. Thus, each policy checkpoint needs to be tried directly on the robot – this can be costly.

Lesson 4: Observation space and hyperparameters play a large role in policy performance.

We found that observation space choice and hyperparameter selection is crucial for good performance. As an example, not including wrist camera observations can reduce performance by 10 to 45 percent

Lesson 5: Using human data for manipulation is promising.

Studying how dataset size impacts performance made us realize that using human data holds much promise. For each task, the bar chart shows how performance changes going from 20% to 50% to 100% of the data. Simpler tasks like Lift and Can require just a fraction of our collected datasets to learn, while more complex tasks like Square and Transport benefit substantially from adding more human data, suggesting that more complex tasks could be addressed by using large human datasets.

Lesson 6: Study results transfer to real world.

We collected 200 demonstrations per task, and trained a BC-RNN policy using identical hyperparameters to simulation, with no hyperparameter tuning. We see that in most cases, performance and insights on what works in simulation transfer well to the real world.

Lift (Real). 96.7% success rate. Nearly matches performance in simulation (100%).
Can (Real). 73.3% success rate. Nearly matches performance in simulation (100%).
Tool Hang (Real). 3.3% success rate. Far from simulation (67.3%) – the real task is harder.

Below, we present examples of policy failures on the Tool Hang task, which illustrate its difficulty, and the large room for improvement.

Insertion Miss
Failed Insertion
Failed Tool Grasp
Tool Drop
Failures which illustrate the difficulty of the Tool Hang task.

We also show that results from our observation space study hold true in the real world – visuomotor policies benefit strongly from wrist observations and pixel shift randomization.

Can (no Wrist). 43.3% success rate (compared to 73.3% with wrist).
Can (no Rand). 26.7% success rate (compared to 73.3% with randomization).
Without wrist observations (left) the success rate decreases from 73.3% to 43.3%. Without pixel shift randomization (right), the success rate decreases from 73.3% to 26.7%.

Takeaways

  1. Learning from large multi-human datasets can be challenging.
  2. Large multi-human datasets hold promise for endowing robots with dexterous manipulation capabilities.
  3. Studying this setting in simulation can enable reproducible evaluation and insights can transfer to real world.

Please see the robomimic website for more information.

This blog post is based on the following paper:

Read More

When do recommender systems amplify user preferences? A theoretical framework and mitigation strategies

What the research is:

Recommender systems have come to influence nearly every aspect of human activity on the internet, whether in the news we read, the products we purchase, or the entertainment we consume. The algorithms and models at the heart of these systems rely on learning our preferences through the course of our interactions with them; when we watch a video or like a post on Facebook, we provide hints to the system about our preferences.

This repeated interplay between people and algorithms creates a feedback loop that results in recommendations that are increasingly customized to our tastes. Ideally, these feedback loops ought to be virtuous all the time; the recommender system is able to infer exactly what our preferences are and provides us with recommendations that enhance the quality of our lives.

However, what happens when the system overindexes and amplifies interactions that do not necessarily capture the user’s true preferences? Or if the user’s preferences have drifted toward recommended items that could be considered harmful or detrimental to their long-term well-being? Under what conditions would recommender systems respond to these changes and amplify preferences leading to a higher prevalence of harmful recommendations?

How it works:

In this paper, we provide a theoretical framework to answer these questions. We model the interactions between users and recommender systems and explore how these interactions may lead to potential harmful outcomes. Our main assumption is that users have a slight inclination to reinforce their opinion (or drift), i.e., increase their preference toward recommendations that they seem to correlate well with, and decrease it otherwise. We characterize the temporal evolution of the user’s preferences as a function of the user, the recommender system, and time, and ask whether this function admits a fixed point, i.e., any change in the system’s response to the user’s interactions does not change their preferences. We show that even under a mild drift and absent any external intervention, no such fixed point exists. That is, even a slight preference by a user for recommendations in a given category can lead to increasingly higher concentrations of item recommendations from that category. We refer to this phenomenon as preference amplification.

Recommender system model

We leverage the well-adopted collaborative filtering model of recommendation systems – each (user, item) tuple receives a score based on the likelihood of the user to be interested in the item. These scores are computed using low-dimension matrix factorization. We use a stochastic recommendation model, in which the set of items presented to a user is chosen probabilistically relative to the items’ scores (rather than deterministically sorting by score). The level of stochasticity in the system is determined by a parameter 𝛽; the higher the 𝛽, the lesser the stochasticity and the distribution of scores is heavily concentrated in the top items. Finally, we think of the content available for recommendation to be benign or problematic, and use ɑ to denote the prevalence of the latter, i.e., the percentage of problematic content out of all content.

Our model also includes the temporal interactions between the user and the recommender system, where in each iteration the user is presented with a set of items, and signals to the recommender system their interests. These interests drift slightly based on the recommended items, the actual magnitude of drift being parameterized by the score the item receives.

The figure below illustrates our temporal drift model. The recommender system initially recommends a diverse set of items to the user, who in turn interacts with those items they prefer. The recommender system picks up this signal, and recommends a less diverse set of items (depicted as only green and blue items) that matches the perceived preferences of the user. The user then drifts further towards a very specific set of items (depicted as the items in blue) that the recommender system suggested. This causes the recommender system to only suggest items from that specific class (blue items).

Simulations

In order to study the parameter space in which the system reinforces recommendation scores, we use simulations with both synthetic and real data sets. We show that the system reinforces scores for items based on the user’s initial preferences — items similar to those liked by the user initially will have a higher likelihood of being recommended over time, and conversely, those that the user did not initially favor will have a decreasing probability of recommendation.

In the figure above on the left, we can see the effect of preference amplification. Solid lines (top group of lines) indicate the likable items, whose probability of receiving a positive reaction from the user is above 0.5. The dashed lines (bottom group) indicate the items that have a low positive reaction from the user. As the figure shows, the probability of liking an item increases toward 1 if its score is positive and toward 0 otherwise. For higher values of 𝛽 (the stochasticity of the recommender system), the stochastic recommender system acts as a Top-N recommender and is therefore more likely to present the users with items that they already liked, leading to stronger reinforcement of their preferences. On the right-side plot in the figure above we see another outcome of preference amplification – the probability of the user liking an item from the top 5% of items recommended to them significantly increases over time. This amplification effect is especially evident for high values of 𝛽, where the stochasticity of the system is low, and the recommender system chooses items that are very likely to be preferred by the user.

Mitigations

Finally, we discuss two strategies for mitigating the effects of preference amplification of problematic entities at a) the global level and b) the personal level. In the former, the strategy is to remove these entities globally in order to reduce their overall prevalence, and in the latter, the system targets users and applies interventions aimed at reducing the probability of recommendation of these entities.


In the figure above we characterize simulation effects of a global intervention on problematic content. We plot the probability of recommending an item of problematic content for different initial prevalences (denoted by ɑ). The figure shows that despite the low prevalence of the problematic content, if there is some initial affinity for that type of content, the probability of it being recommended to the user increases over time.

In the paper, we also describe an experiment we conducted using a real-world large-scale video recommender system. In the experiment, we downrank videos considered to include borderline nudity (the platform already filters out videos that violate community standards) for users who have a high level of exposure to them consistently. The results of the experiment show that in addition to reducing exposure of this content in the impacted population, we saw that overall engagement go up by +2%. These results are highly encouraging, as not only we can prevent exposure to problematic content, we also have an overall positive effect on the user experience.

Why it matters:

In this work, we study the interactions between users and recommender systems, and show that for certain user behaviors, their preferences can be amplified by the recommender system. Understanding the long-term impact of ML systems helps us, as practitioners, to build better safeguards and ensure that our models are optimized to serve the best interests of our users.

Read the full paper:

A framework for understanding preference amplification in recommender systems

Learn More:

Watch our presentation at KDD 2021.

The post When do recommender systems amplify user preferences? A theoretical framework and mitigation strategies appeared first on Facebook Research.

Read More

Make your audio and video files searchable using Amazon Transcribe and Amazon Kendra

The demand for audio and video media content is growing at an unprecedented rate. Organizations are using media to engage with their audiences like never before. Product documentation is increasingly published in video form, and podcasts are increasingly produced in place of blog posts. The recent explosion in the use of virtual workplaces has resulted in content that is encapsulated in the form of recorded meetings, calls, and voicemails. Contact centers also generate media content such as support calls, screen share recordings, or post-call surveys.

Amazon Machine Learning services help you find answers and extract valuable insights from the content of your audio and video files as well as your text files.

In this post, we introduce a new open-source solution, MediaSearch, built to make your media files searchable, and consumable in search results. It uses Amazon Transcribe to convert media audio tracks to text, and Amazon Kendra to provide intelligent search. Your users can find the content they’re looking for, even when it’s embedded in the sound track of your audio or video files. The solution also provides an enhanced Amazon Kendra query application that lets users play the relevant section of original media files, directly from the search results page.

Solution overview

MediaSearch is easy to install and try out! Use it to enable your customers to find answers to their questions from your podcast recordings and presentations, or for your students to find answers from your educational videos or lecture recordings, in addition to text documents.

The MediaSearch solution has two components, as illustrated in the following diagram.

The first component, the MediaSearch indexer, finds and transcribes audio and video files stored in an Amazon Simple Storage Service (Amazon S3) bucket. It prepares the transcriptions by embedding time markers at the start of each sentence, and it indexes each prepared transcription in a new or existing Amazon Kendra index. It runs the first time when you install it, and subsequently runs on an interval that you specify, maintaining the index to reflect any new, modified, or deleted files.

The second component, the MediaSearch finder, is a sample web search client that you use to search for content in your Amazon Kendra index. It has all the features of a standard Amazon Kendra search page, but it also includes in-line embedded media players in the search result, so you can not only see the relevant section of the transcript, but also play the corresponding section from the original media without navigating away from the search page (see the following screenshot).

In the sections that follow, we discuss several topics:

  • How to deploy the solution to your AWS account
  • How to use it to index and search sample media files
  • How to use the solution with your own media files
  • How the solution works under the hood
  • The costs involved
  • How to monitor usage and troubleshoot problems
  • Options to customize and tune the solution
  • How to uninstall and clean up when you’re done experimenting

Deploy the MediaSearch solution

In this section, we walk through deploying the two solution components: the indexer and the finder. We use an AWS CloudFormation stack to deploy the necessary resources in the us-east-1 (N. Virginia) AWS Region.

The source code is available in our GitHub repository. Follow the directions in the README to deploy MediaSearch to additional Regions supported by Amazon Kendra.

Deploy the indexer component

To deploy the indexer component, complete the following steps:

  1. Choose Launch Stack:
  2. Change the stack name if required to ensure that it’s unique.
  3. For ExistingIndexId, leave blank to create a new Amazon Kendra index (Developer Edition), otherwise provide the IndexId (not the index name) for an existing index in your account and Region (Amazon Kendra Enterprise Edition should be used for production workloads).
  4. For MediaBucket and MediaFolderPrefix, use the defaults initially to transcribe and index sample audio and video files.
  5. For now, use the default values for the other parameters.
  6. Select the acknowledgement check boxes, and choose Create stack.
  7. When the stack is created (after approximately 15 minutes), choose the Outputs tab, and copy the value of IndexId—you need it to deploy the finder component in the next step.

The newly installed indexer runs automatically to find, transcribe, and index the sample audio and video files. Later you can provide a different bucket name and prefix to index your own media files. If you have media files in multiple buckets, you can deploy multiple instances of the indexer, each with a unique stack name.

Deploy the finder component

To deploy the finder web application component, complete the following steps:

  1. Choose Launch Stack:
  2. For IndexId, use the Amazon Kendra index copied from the MediaSearch indexer stack outputs.
  3. For MediaBucketNames, use the default initially to allow the search page to access media files from the sample file bucket.
  4. When the stack is created (after approximately 5 minutes), choose the Outputs tab and use the link for MediaSearchFinderURL to open the new media search application page in your browser.

If the application isn’t ready when you first open the page, don’t worry! The initial application build and deployment (using AWS Amplify) takes about 10 minutes, so it will work when you try again a little later. If for any reason the application still doesn’t open, refer to the README in the GitHub repo for troubleshooting steps.

And that’s all there is to the deployment! Next, let’s run some search queries to see it in action.

Test with the sample media files

By the time the MediaSearch finder application is deployed and ready to use, the indexer should have completed processing the sample media files (selected AWS Podcast episodes and AWS Knowledge center videos). You can now run your first MediaSearch query.

  1. Open the MediaSearch finder application in your browser as described in the previous section.
  2. In the query box, enter What’s an interface VPC Endpoint?

The query returns multiple results, sourced from the transcripts of the sample media files.

  1. Observe the time markers at the beginning of each sentence in the answer text. This indicates where the answer is to be found in the original media file.
  2. Use the embedded video player to play the original video inline. Observe that the media playback starts at the relevant section of the video based on the time marker.
  3. To play the video full screen in a new browser tab, use the Fullscreen menu option in the player, or choose the media file hyperlink shown above the answer text.
  4. Choose the video file hyperlink (right-click), copy the URL, and paste it into a text editor. It looks something like the following:
https://mediasearchtest.s3.amazonaws.com/mediasamples/What_is_an_Interface_VPC_Endpoint_and_how_can_I_create_Interface_Endpoint_for_my_VPC_.mp4?AWSAccessKeyId=ASIAXMBGHMGZLSYWJHGD&Expires=1625526197&Signature=BYeOXOzT585ntoXLDoftkfS4dBU%3D&x-amz-security-token=.... #t=253.52

This is a presigned S3 URL that provides your browser with temporary read access to the media file referenced in the search result. Using presigned URLs means you don’t need to provide permanent public access to all of your indexed media files.

  1. Scroll down the page, and observe that some search results are from audio (MP3) files, and some are from video (MP4) files.

You can mix and match media types in the same index. You could include other data source types as well, such as documents, webpages, and other file types supported by available Amazon Kendra data sources, and search across them all, allowing Amazon Kendra to find the best content to answer your query.

  1. Experiment with additional queries, such as What does a solutions architect do? or What is Kendra?, or try your own questions.

Index and search your own media files

To index media files stored in your own S3 bucket, replace the MediaBucket and MediaFolderPrefix parameters with your own bucket name and prefix when you install or update the indexer component stack, and modify the MediaBucketNames parameter with your own bucket name when you install or update the finder component stack.

  1. Create a new MediaSearch indexer stack using an existing Amazon Kendra IndexId to add files stored in the new location. To deploy a new indexer, follow the directions in the Deploy the indexer component section in this post, but this time replace the defaults to specify the media bucket name and prefix for your own media files.
  2. Alternatively, update an existing MediaSearch indexer stack to replace the previously indexed files with files from the new location:
    1. Select the stack on the CloudFormation console, choose Update, then Use current template, then Next.
    2. Modify the media bucket name and prefix parameter values as needed.
    3. Choose Next twice, select the acknowledgement check box, and choose Update stack.
  3. Update an existing MediaSearch finder stack to change bucket names or add additional bucket names to the MediaBucketNames parameter.

When the MediaSearch indexer stack is successfully created or updated, the indexer automatically finds, transcribes, and indexes the media files stored in your S3 bucket. When it’s complete, you can submit queries and find answers from the audio tracks of your own audio and video files.

You have the option to provide metadata for any or all of your media files. Use metadata to assign values to index attributes for sorting, filtering, and faceting your search results, or to specify access control lists to govern access to the files. Metadata files can be in the same S3 folder as your media files (default), or in a parallel folder structure specified by the optional indexer parameter MetadataFolderPrefix. For more information about how to create metadata files, see S3 document metadata.

You can also provide customized transcription options for any or all of your media files. This allows you to take full advantage of Amazon Transcribe features such as custom vocabularies, automatic content redaction, and custom language models. For more information, refer to the README in the GitHub repo.

How the MediaSearch solution works

Let’s take a quick look under the hood to see how the solution works, as illustrated in the following diagram.

The MediaSearch solution has an event-driven serverless computing architecture with the following steps:

  1. You provide an S3 bucket containing the audio and video files you want to index and search.
  2. Amazon EventBridge generates events on a repeating interval (such as every 2 hours, every 6 hours, and so on)
  3. These events invoke an AWS Lambda function. The function is invoked initially when the CloudFormation stack is first deployed, and then subsequently by the scheduled events from EventBridge. An Amazon Kendra data source sync job is started. The Lambda function lists all the supported media files (FLAC, MP3, MP4, Ogg, WebM, AMR, or WAV) and associated metadata and Transcribe options stored in the user-provided S3 bucket.
  4. Each new file is added to the Amazon DynamoDB tracking table and submitted to be transcribed by a Transcribe job. Any file that has been previously transcribed is submitted for transcription again only if it has been modified since it was previously transcribed, or if associated Transcribe options have been updated. The DynamoDB table is updated to reflect the transcription status and last modified timestamp of each file. Any tracked files that no longer exist in the S3 bucket are removed from the DynamoDB table and from the Amazon Kendra index. If no new or updated files are discovered, the Amazon Kendra data source sync job is immediately stopped. The DynamoDB table holds a record for each media file with attributes to track transcription job names and status, and last modified timestamps.
  5. As each Transcribe job completes, EventBridge generates a Job Complete event, which invokes an instance of another Lambda function.
  6. The Lambda function processes the transcription job output, generating a modified transcription that has a time marker inserted at the start of each sentence. This modified transcription is indexed in Amazon Kendra, and the job status for the file is updated in the DynamoDB table. When the last file has been transcribed and indexed, the Amazon Kendra data source sync job is stopped.
  7. The index is populated and kept in sync with the transcriptions of all the media files in the S3 bucket monitored by the MediaSearch indexer component, integrated with any additional content from any other provisioned data sources. The media transcriptions are used by Amazon Kendra’s intelligent query processing, which allows users to find content and answers to their questions.
  8. The sample finder client application enhances users’ search experience by embedding an inline media player with each Amazon Kendra answer that is based on a transcribed media file. The client uses the time markers embedded in the transcript to start media playback at the relevant section of the original media file.

Estimate costs

In addition to Amazon S3 costs associated with storing your media, the MediaSearch solution incurs usage costs from Amazon Kendra and Transcribe. Additional minor (usually not significant) costs are incurred by the other services mentioned after free tier allowances have been used. For more information, see the pricing documentation for Amazon Kendra, Transcribe, Lambda, DynamoDB, and EventBridge.

Pricing example: Index the sample media files

The sample dataset has 25 media files—13 audio podcast and 12 video files—containing a total of around 480 minutes or 29,000 seconds of audio.

If you don’t provide an existing Amazon Kendra IndexId when you install MediaSearch, a new Amazon Kendra Developer Edition index is automatically created for you so you can test the solution. After you use your free tier allowance (up to 750 hours in the first 30 days), the index costs $1.125 per hour.

Transcribe pricing is based on the number of seconds of audio transcribed, with a free tier allowance of 60 minutes of audio per month for the first 12 months. After the free tier is used, the cost is $0.00040 for each second of audio transcribed. If you’re no longer free tier eligible, the cost to transcribe the sample files is as follows:

  • Total seconds of audio = 29,000
  • Transcription price per second = $0.00040
  • Total cost for Transcribe = [number of seconds] x [cost per second] = 29,000 x $0.00040 = $11.60

Monitor and troubleshoot

To see the details of each media file transcript job, navigate to the Transcription jobs page on the Transcribe console.

Each media file is transcribed only one time, unless the file is modified. Modified files are re-transcribed and re-indexed to reflect the changes.

Choose any transcription job to review the transcription and examine additional job details.

On the Indexes page of the Amazon Kendra console, choose the index used by MediaSearch to examine the index details.

Choose Data sources in the navigation pane to examine the MediaSearch indexer data source, and observe the data source sync run history. The data source syncs when the indexer runs every interval specified in the CloudFormation stack parameters when you deployed or last updated the solution.

On the DynamoDB console, choose Tables in the navigation pane. Use your MediaSearch stack name as a filter to display the MediaSearch DynamoDB table, and examine the items showing each indexed media file and corresponding status. The table has one record for each media file, and contains attributes with information about the file and its processing status.

On the Functions page of the Lambda console, use your MediaSearch stack name as a filter to list the two MediaSearch indexer functions described earlier.

Choose either of the functions to examine the function details, including environment variables, source code, and more. Choose Monitor & View logs in CloudWatch to examine the output of each function invocation and troubleshoot any issues.

Customize and enhance the solution

You can fork the MediaSearch GitHub repository, enhance the code, and send us pull requests so we can incorporate and share your improvements!

The following are a few suggestions for features you might want to implement:

Clean up

When you’re finished experimenting with this solution, clean up your resources by using the AWS CloudFormation console to delete the indexer and finder stacks that you deployed. This deletes all the resources, including any Amazon Kendra indexes that were created by deploying the solution. Pre-existing indexes aren’t deleted. However, media files that were indexed by the solution are removed from the pre-existing index when you delete the indexer stack.

Conclusion

The combination of Amazon Transcribe and Amazon Kendra enable a scalable, cost-effective solution to make your media files discoverable. You can use the content of your media files to find accurate answers to your users’ questions, whether they’re from text documents or media files, and consume them in their native format. In other words, this solution is a leap in bringing media files on par with text documents as containers of information.

The sample MediaSearch application is provided as open source—use it as a starting point for your own solution, and help us make it better by contributing back fixes and features via GitHub pull requests. For expert assistance, AWS Professional Services and other Amazon partners are here to help.

We’d love to hear from you. Let us know what you think in the comments section, or using the issues forum in the MediaSearch GitHub repository.

About the Authors

Bob StrahanBob Strahan is a Principal Solutions Architect in the AWS Language AI Services team.

 

 

 

 

Abhinav JawadekarAbhinav Jawadekar is a Senior Partner Solutions Architect at Amazon Web Services. Abhinav works with AWS Partners to help them in their cloud journey.

 

Read More

Detect anomalies in operational metrics using Dynatrace and Amazon Lookout for Metrics

Organizations of all sizes and across all industries gather and analyze metrics or key performance indicators (KPIs) to help their businesses run effectively and efficiently. Operational metrics are used to evaluate performance, compare results, and track relevant data to improve business outcomes. For example, you can use operational metrics to determine application performance (the average time it takes to render a page for an end user) or application availability (the duration of time the application was operational). One challenge that most organizations face today is detecting anomalies in operational metrics, which are key in ensuring continuity of IT system operations.

Traditional rule-based methods are manual and look for data that falls outside of numerical ranges that have been arbitrarily defined. An example of this is an alert when transactions per hour fall below a certain number. This results in false alarms if the range is too narrow, or missed anomalies if the range is too broad. These ranges are also static. They don’t change based on evolving conditions like the time of the day, day of the week, seasons, or business cycles. When anomalies are detected, developers, analysts, and business owners can spend weeks trying to identify the root cause of the change before they can take action.

Amazon Lookout for Metrics uses machine learning (ML) to automatically detect and diagnose anomalies without any prior ML experience. In a couple of clicks, you can connect Lookout for Metrics to popular data stores like Amazon Simple Storage Service (Amazon S3), Amazon Redshift, and Amazon Relational Database Service (Amazon RDS), as well as third-party software as a service (SaaS) applications (such as Salesforce, Dynatrace, Marketo, Zendesk, and ServiceNow) via Amazon AppFlow and start monitoring metrics that are important to your business.

This post demonstrates how you can connect to your IT operational infrastructure monitored by Dynatrace using Amazon AppFlow and set up an accurate anomaly detector across metrics and dimensions using Lookout for Metrics. The solution allows you to set up a continuous anomaly detector and optionally set up alerts to receive notifications when anomalies occur.

Lookout for Metrics integrates seamlessly with Dynatrace to detect anomalies within your operational metrics. Once connected, Lookout for Metrics uses ML to start monitoring data and metrics for anomalies and deviations from the norm. Dynatrace enables monitoring of your entire infrastructure, including your hosts, processes, and network. You can perform log monitoring and view information such as the total traffic of your network, the CPU usage of your hosts, the response time of your processes, and more.

Amazon AppFlow is a fully managed service that provides integration capabilities by enabling you to transfer data between SaaS applications like Datadog, Salesforce, Marketo, and Slack and AWS services like Amazon S3 and Amazon Redshift. It provides capabilities to transform, filter, and validate data to generate enriched and usable data in a few easy steps.

Solution overview

In this post, we demonstrate how to integrate with an environment monitored by Dynatrace and detect anomalies in the operation metrics. We also determine how application availability and performance (resource contention) were impacted.

The source data is a cluster of Amazon Elastic Compute Cloud (Amazon EC2) instances that is monitored by Dynatrace. Each EC2 instance is installed with Dynatrace OneAgent to collect all monitored telemetry data (CPU utilization, memory, network utilization, and disk I/O). Amazon AppFlow enables you to securely integrate SaaS applications like Dynatrace and automate data flows, while providing options to configure and connect to such services natively from the AWS Management Console or via API. In this post, we focus on connecting to Dynatrace as our source and Lookout for Metrics as the target, both of which are natively supported applications in Amazon AppFlow.

The solution enables you to create an Amazon AppFlow data flow from Dynatrace to Lookout for Metrics. You can then use Lookout for Metrics to detect any anomalies in the telemetry data, as shown in the following diagram. Optionally, you can send automated anomaly alerts to AWS Lambda functions, webhooks, or Amazon Simple Notification Service (Amazon SNS) topics.

The following are the high-level steps to implement the solution:

  1. Set up Amazon AppFlow integration with Dynatrace.
  2. Create an anomaly detector with Lookout for Metrics.
  3. Add a dataset to the detector and integrate Dynatrace metrics.
  4. Activate the detector.
  5. Create an alert.
  6. Review the detector and data flow status.
  7. Review and analyze any anomalies.

Set up Amazon AppFlow integration with Dynatrace

To set up the data flow, complete the following steps:

  1. On the Amazon AppFlow console, choose Create flow.
  2. For Flow name, enter a name.
  3. For Flow description, enter an optional description.
  4. In the Data encryption section, you can choose or create an AWS Key Management Service (AWS KMS) key.
  5. Choose Next.
  6. For Source name, choose Dynatrace.
  7. For Choose Dynatrace Connection, choose the connection you created.
  8. For Choose Dynatrace object, choose Problems (this is the only object supported as of this writing).

For more information about Dynatrace problems, see Problem overview page.

  1. For Destination name, choose Amazon Lookout for Metrics.
  2. For API token, generate an API token from the Dynatrace console.
  3. For Subdomain, enter your Dynatrace portal URL address.
  4. For Data encryption, choose the AWS KMS key.
  5. For Connection Name, enter a name.
  6. Choose Connect.
  7. For Flow trigger, select Run flow on schedule.
  8. For Repeats, choose Minutes (alternatively, you can choose hourly or daily).
  9. Set the trigger to repeat every 5 minutes.
  10. Enter a starting time.
  11. Enter a start date.

Dynatrace requires a between date range filter to be set.

  1. For Field name, choose Date range.
  2. For Condition, choose is between.
  3. For Criteria 1, choose your start date.
  4. For Criteria 2, choose your end date.
  5. Review your settings and choose Create flow.

Create an anomaly detector with Lookout for Metrics

To create your anomaly detector, complete the following steps:

  1. On the Lookout for Metrics console, choose Create detector.
  2. For Detector name, enter a name.
  3. For Description, enter an optional description.
  4. For Interval, choose the time between each analysis. This should match the interval set on the flow.
  5. For Encryption, create or choose an existing AWS KMS key.
  6. Choose Create.

Add a dataset to the detector and integrate Dynatrace metrics

The next step in activating your anomaly detector is to add a dataset and integrate the Dynatrace metrics.

  1. On the detector details, choose Add a dataset.
  2. For Name, enter the data source name.
  3. For Description, enter an optional description.
  4. For Timezone, choose the time zone relevant to your dataset. This should match the time zone used in Amazon AppFlow (which picks up from the browser).
  5. For Datasource, choose Dynatrace.
  6. For Amazon AppFlow flow, choose the flow that you created.
  7. For Permissions, choose a service role.
  8. Choose Next.
  9. For Map fields, the detector tracks 5 measures; in this example I choose impactLevel and hasRootCause.

The map fields are the primary fields that the detector monitors. The fields that are relevant to monitor from an operational KPI should be considered.

  1. For Dimensions, the detector creates segments in measure values. For this post, I choose severityLevel.
  2. Review the settings and choose Save dataset.

Activate the detector

You’re now ready to activate the newly created detector.

Create an alert

You can create an alert to send automated anomaly alerts to Lambda functions; webhooks; cloud applications like Slack, PagerDuty, and DataDog; or to SNS topics with subscribers that use SMS, email, or push notifications.

  1. On the detector details, choose Add alerts.
  2. For Alert Name, enter the name.
  3. For Sensitivity threshold, enter a threshold at which the detector sends anomaly alerts.
  4. For Channel, choose either Amazon SNS or Lambda as the notification method. For this post, I use Amazon SNS.
  5. For SNS topic, create or choose an existing SNS topic.
  6. For Service role, choose an execution role.
  7. Choose Add alert.

Review the detector and flow status

On the Run history tab, you can confirm that the flows are running successfully for the interval chosen.

On the Detector log tab, you can confirm that the detector records the results after each interval.

Review and analyze any anomalies

On the main detector page, choose View anomalies to review and analyze any anomalies.

On the Anomalies page, you can adjust the severity score on the threshold dial to filter anomalies above a given score.

The following analysis represents the severity level and impacted metrics. The graph suggests anomalies detected by the detector with the availability and resource contention being impacted. The anomaly was detected on June 28 at 14:30 PDT and has a severity score of 98, indicating a high severity anomaly that needs immediate attention.

Lookout for Metrics also allows you to provide real-time feedback on the relevance of the detected anomalies, which enables a powerful human-in-the-loop mechanism. This information is fed back to the anomaly detection model to improve its accuracy continuously, in near-real time.

Conclusion

Anomaly detection can be very useful in identifying anomalies that could signal potential issues within your operational environment. Timely detection of anomalies can aid in troubleshooting, help avoid loss in revenue, and help maintain your company’s reputation. Lookout for Metrics automatically inspects and prepares the data, selects the best-suited ML algorithm, begins detecting anomalies, groups related anomalies together, and summarizes potential root causes.

To get started with this capability, see Amazon Lookout for Metrics. You can use this capability in all Regions where Lookout for Metrics is publicly available. For more information about Region availability, see AWS Regional Services.

About the Author

Sumeeth Siriyur is a Solutions Architect based out of AWS, Sydney. He is passionate about infrastructure services and uses AI services to influence IT infrastructure observability and management. In his spare time, he likes binge-watching and works to continually improve his outdoor sports.

Read More

Accenture promotes machine learning growth with world’s largest private AWS DeepComposer Battle of the Bands League

Accenture is known for pioneering innovative solutions to achieve customer success by using artificial intelligence (AI) and machine learning (ML) powered solutions with AWS services. To keep teams updated with latest ML services, Accenture seeks to gamify hands-on learning. One such event, AWS DeepComposer Battle of the Bands, hosted by Accenture, is the world’s first and largest global league.

Accenture’s league spanned 16 global regions and 55 countries, with each location competing for global superstardom and a real-life gold record! With around 500 bands in the competition, Accenture employees from different skills and domain knowledge proved themselves as aspiring ML musicians, generating a playlist of 150 original songs using AWS DeepComposer. There was no shortage of fans either, with thousands of votes being cast by supportive colleagues and teammates.

Why an AWS DeepComposer Battle of Bands and why now?

According to a recent Gartner report, “Despite the global impact of COVID-19, 47% of AI investments were unchanged since the start of the pandemic and 30% of organizations actually planned to increase such investments”. Additionally, there have been few opportunities in this pandemic to share a fun and enjoyable experience with our teammates, let alone colleagues around the globe.

Accenture and their Amazon Business Group are always looking for unique and exciting ways to help employees up-skill in the latest and greatest tech. Being inspired by their massively successful annual AWS DeepRacer Grand Prix, Accenture switched out the racetrack for the big stage and created their own Battle of the Bands using AWS DeepComposer.

This Battle of the Bands brought together fans and bands from around the globe, generating thousands of views, shares, votes, and opportunities to connect, laugh, and smile together.

Education was Accenture’s number one priority when crafting the competition. The goal was to expose those unfamiliar with AWS or ML to a fun and approachable experience that would increase their confidence with this technology and start them down a path of greater learning. According to registration metrics, around half of all participants were working with AWS and ML hands-on for the first time. Participants have shared that this competition inspired them to learn more about both AWS and ML. Some feedback received included:

“I enjoyed doing something creative and tackling music, which I had no experience with previously.”

“It was fun trying to make a song with the tool and to learn about other ML techniques.”

“I was able to feel like a musician even though I don’t know much about music composition.”

A hall of fame alliance

Accenture and AWS have always demonstrated a great alliance. In 2019, Accenture hosted one of the world’s largest private AWS DeepRacer Leagues. In 2021, multiple individuals and groups participated in the AWS DeepComposer Battle of the Bands League. These bands were able to create a video to go along with their song submission, allowing for more creative freedom and a chance to stand out from the crowd. Some bands made artistic music videos, others saw an opportunity to make something funny and share laughs around the world. Going above and beyond, one contestant turned their AWS DeepComposer competition into a sing-along training video for Accenture’s core values, while another dedicated their video to honoring “sheroes” and famous women in tech.

The dedication of Accenture’s bands to the spirit of the competition really showed in the array of pun-filled band names such as “Doggo-as-a-service,” “The Oracles,” “Anna and the AlgoRhythms,” and “#000000 Sabbath.”

AWS offers a portfolio of educational devices—AWS DeepLens, AWS DeepRacer, and AWS DeepComposer—designed for developers of all skill levels to learn the fundamentals of ML in fun and practical ways. The hands-on nature of AWS AI devices makes them great tools to engage and educate employees.

Accelerating innovation with the Accenture AWS Business Group

By working with the Accenture AWS Business Group (AABG), you can learn from the resources, technical expertise, and industry knowledge of two leading innovators, helping you accelerate the pace of innovation to deliver disruptive products and services. The AABG helps you ideate and innovate cloud solutions through rapid prototype development.

Connect with our team at accentureaws@amazon.com to learn how to use and accelerate ML in your products and services.

You can also organize your own event. To learn more about AWS DeepComposer events, see AWS DeepRacer Community Blog and also check out blog on How to run an AI powered musical challenge: “AWS DeepComposer Got Talent” to learn more about how to host your first event with AWS DeepComposer.

About Accenture

Accenture is a global professional services company with leading capabilities in digital, cloud and security. Combining unmatched experience and specialized skills across more than 40 industries, we offer Strategy and Consulting, Interactive, Technology and Operations services — all powered by the world’s largest network of Advanced Technology and Intelligent Operations centers. Our 569,000 people deliver on the promise of technology and human ingenuity every day, serving clients in more than 120 countries. We embrace the power of change to create value and shared success for our clients, people, shareholders, partners and communities. Visit us at www.accenture.com.

Copyright © 2021 Accenture. All rights reserved. Accenture and its logo are trademarks of Accenture.

This document is produced by consultants at Accenture as general guidance. It is not intended to provide specific advice on your circumstances. If you require advice or further details on any matters referred to, please contact your Accenture representative.

This document makes descriptive reference to trademarks that may be owned by others. The use of such trademarks herein is not an assertion of ownership of such trademarks by Accenture and is not intended to represent or imply the existence of an association between Accenture and the lawful owners of such trademarks. No sponsorship, endorsement, or approval of this content by the owners of such trademarks is intended, expressed, or implied.

Accenture provides the information on an “as-is” basis without representation or warranty and accepts no liability for any action or failure to act taken in response to the information contained or referenced in this publication.

About the Authors

Marc DeMory is a senior emerging tech consultant with Accenture’s Chicago Liquid Studio, focusing on rapid-prototyping and cloud-native development in the fields of Machine Learning, Computer Vision, Automation, and Extended Reality.

 

 

 

Sameer Goel is a Sr. Solutions Architect in Netherlands, who drives customer success by building prototypes on cutting-edge initiatives. Prior to joining AWS, Sameer graduated with a master’s degree from NEU Boston, with a concentration in data science. He enjoys building and experimenting with AI/ML projects on Raspberry Pi.

 

 

Maryam rezapoor is a Senior Product Manager with AWS DeepLabs team based in Santa Clara, CA. She works on developing products to put Machine Learning in the hands of everyone. She loves hiking through the US national parks and is currently training for 1-day Grand Canyon Rim to Rim hike. She is a fan of Metallica and Evanescence. The drummer, Lars Ulrich, has inspired her to pick up those sticks and play drum while singing “nothing else matters.”

Read More