Mango User Guide

Introduction

Mango is a distributed visualization tool that enables visualization of genomic data on top of Apache Spark.

The Mango/Big Data Genomics Ecosystem

Mango builds upon the open source Apache Spark, Apache Avro, and Apache Parquet projects. Additionally, Mango can be deployed for both interactive and production workflows using a variety of platforms.

Installation

• Building Mango from Source
  • Running Mango
  • Building for Python
• Installing Mango from Conda
  • Install Conda
  • Set up channels
  • Install Mango in a conda environment
• Running Mango from Distribution
  • Fetching Mango Distribution
  • Installing python modules

Usage and Examples

• Jupyter Widget Usage
  • Installation for Jupyter notebook
  • Installation for Jupyter lab
  • Pileup Example
  • Variant Example
  • Feature Example
  • Genotype Example
• Mango Browser Examples
  • Mango Browser Options
  • Running Mango Browser Examples Locally
  • Running Mango Browser with Parameters
  • Running example files on a cluster with HDFS
  • Running on Apache YARN
• Building a Genome
  • Assembling a Default Genome File
  • Assembling a Custom Genome File
• Mango Python Examples
  • Running Mango Notebook Locally
  • Running the Mango Notebook with Parameters
  • Running the Mango Notebook on YARN

API

• Mango Pyspark API Documentation
  • bdgenomics.mango Package
    • Alignments
    • Coverage
    • Features
    • Variants
    • Genotypes
• Jupyter Widget API Documentation
  • bdgenomics.mango.pileup Package
    • PileupViewer
    • Sources
    • Track

Supported File Types

• Supported Files
  • Supported File Types
  • Accessing http files through Mango
  • Accessing s3a files through Mango

Running with Docker

• Running Mango from Docker
  • Running Mango Browser on Docker
  • Running Mango Notebook on Docker
  • Creating a Mango genome using Docker

Google Cloud

• Running Mango on Google Cloud
  • Before you Start
  • Creating a Dataproc Cluster
• Running Mango on GCE through Docker
  • Running the Mango Notebook on GCE
  • Running the Mango Browser on GCE

Amazon EMR

• Running Mango from Amazon EMR
  • Before you Start
• Running Mango through Docker
  • Creating a Cluster
  • Enabling a Web Connection
  • Connecting to Cluster
  • Accessing the Web UI
  • Testing your Configuration
  • Running the Mango Browser on EMR with Docker
  • Creating a reference genome on EMR with Docker
  • Running Mango Notebook on EMR with Docker
  • Accessing files in the Mango notebook from HDFS
• Running Mango Standalone
  • Creating a Cluster
  • Running Mango Browser on EMR
  • Running Mango Notebook on EMR

Development

• Development notes for the Mango Browser
  • Debugging the Mango browser frontend

• Index
• Search Page

References

Massie, Matt, Frank Nothaft, Christopher Hartl, Christos Kozanitis, André Schumacher, Anthony D Joseph, and David A Patterson. 2013. “ADAM: Genomics Formats and Processing Patterns for Cloud Scale Computing.” UCB/EECS-2013-207, EECS Department, University of California, Berkeley.

Morrow, Alyssa, Anthony Joseph, and Nir Yosef. 2017. “Distributed Visualization for Genomic Analysis.” UCB/EECS-2017-82, EECS Department, University of California, Berkeley.

McKenna, Aaron, Matthew Hanna, Eric Banks, Andrey Sivachenko, Kristian Cibulskis, Andrew Kernytsky, Kiran Garimella, et al. 2010. “The Genome Analysis Toolkit: A MapReduce Framework for Analyzing Next-Generation DNA Sequencing Data.” Genome Research 20 (9). Cold Spring Harbor Lab: 1297–1303.

Melnik, Sergey, Andrey Gubarev, Jing Jing Long, Geoffrey Romer, Shiva Shivakumar, Matt Tolton, and Theo Vassilakis. 2010. “Dremel: Interactive Analysis of Web-Scale Datasets.” Proceedings of the VLDB Endowment 3 (1-2). VLDB Endowment: 330–39.

Nothaft, Frank A, Matt Massie, Timothy Danford, Zhao Zhang, Uri Laserson, Carl Yeksigian, Jey Kottalam, et al. 2015. “Rethinking Data-Intensive Science Using Scalable Analytics Systems.” In Proceedings of the 2015 ACM SIGMOD International Conference on Management of Data (SIGMOD ’15). ACM.

Schadt, Eric E, Michael D Linderman, Jon Sorenson, Lawrence Lee, and Garry P Nolan. 2010. “Computational Solutions to Large-Scale Data Management and Analysis.” Nature Reviews Genetics 11 (9). Nature Publishing Group: 647–57.

Vanderkam, Dan, B. Arman Aksoy, Isaac Hodes, Jaclyn Perrone, and Jeff Hammerbacher. 2016. “pileup.js: a JavaScript library for interactive and in-browser visualization of genomic data.” In Bioinformatics, 32 (15).

Vavilapalli, Vinod Kumar, Arun C Murthy, Chris Douglas, Sharad Agarwal, Mahadev Konar, Robert Evans, Thomas Graves, et al. 2013. “Apache Hadoop YARN: Yet Another Resource Negotiator.” In Proceedings of the Symposium on Cloud Computing (SoCC ’13), 5. ACM.

Zaharia, Matei, Mosharaf Chowdhury, Tathagata Das, Ankur Dave, Justin Ma, Murphy McCauley, Michael Franklin, Scott Shenker, and Ion Stoica. 2012. “Resilient Distributed Datasets: A Fault-Tolerant Abstraction for in-Memory Cluster Computing.” In Proceedings of the Conference on Networked Systems Design and Implementation (NSDI ’12), 2. USENIX Association.