swayable-data documentation

Version v0.114

---

Swayable Data is responsible for all calculations and analyses behind the Swayable product. Swayable Data also supports analyzing or reanalyzing data via a command line interface or a Jupyter notebook.

Swayable Data uses Pandas, Numpy, and Scipy for calculations. In production, Swayable Data runs as a Celery daemon using Redis as a broker. A Flask-based API is used to trigger calculations in Celery and to download data. Data storage in MongoDB is supported.

Structure Overview

Swayable Data is divided into five main packages, plus the command line interface defined in the sway package.

Analysis

The analysis package is responsible for all substantial calculations, model training, and model-based predictions. The module is completely decoupled from the other packages and never imports them. This allows these routines to be called on data that is not loaded from our database e.g. in a notebook with data from a spreadsheet.

API

The api package defines a lightweight web API using Flask. This API is called by our web applications to trigger analyses and download raw or analyzed data.

App

The app package is where the Celery interface is defined. The tasks that run within the Celery daemon are defined in app.tasks. Several tasks run on a schedule and are not called by any other part of the production stack. The schedule is defined in app.app. Tasks are also triggered in production in response to interactions in the UI. This is done via calls to the api package. Lastly, tasks can be triggered in productiomn via the command-line interface in the sway package.

IO

The io package is where the interfaces to all supporting services are called. Currently, these include MongoDB, slack, and the U.S Census API. The preferred module for reading and writing to our MongoDB is io.dao [TODO: change to a proper link]. This uses a DAO-style architecture to fully abstract that the source of the data is MongoDB. In the future, we may expand to connect to other types of databases. The io.mongo package is the legacy interface to MongoDB. We are currently transitioning away from using this module for any interactions with our MongoDB.

Pipe

The pipe package module manages the pipeline of analysis including reading data and configuration from the datasources, calling the appropriate analysis functions, and writing the answers to the database.

Sway

The sway package is a command line interface with options to analyze a survey or download data as CSVs.

Contents

• How Tos
• API Reference
  • analysis package
    • analysis.calculated_fields package
    • analysis.calculated_fields.geodemographics module
    • analysis.dimensions module
    • analysis.frequentist module
    • analysis.indices module
    • analysis.meta module
    • analysis.nlp module
    • analysis.poststrat module
    • analysis.preprocess module
    • analysis.pretest module
    • Subpackages
      • analysis.models package
  • api package
    • api.auth module
    • api.routes module
  • app package
    • app.app module
    • app.tasks module
    • app.tasks.analysis module
    • app.tasks.models module
    • app.tasks.preprocess module
  • io package
    • Subpackages
      • io.apis package
      • io.dao package
      • io.mongo package
    • io.slack module
  • pipe package
    • pipe.analysis module
    • pipe.census module
    • pipe.models module
    • pipe.preprocess module
    • pipe.quality module
  • sway package
  • utilities package

Indices

• Index

• Module Index