• Python
• Multi-Mechanize
• Jenkins-CI

Corey Goldberg
Boston Python User Group
Dec 19, 2011 - Cambridge, MA

• Slides: http://tinyurl.com/bpwebperf

• Lives/Works in Boston
• Performance Testing since 1999
• Open Source Hacker
• Prefers Python
• Works for Canonical in Core Dev Ops
• Nothing to Pitch

• launching a new application
• tuning existing services
• preparing for future capacity
• changing IT infrastructure
• making everything "web scale"

• Server/System Performance
• Network/Infrastructure Performance
• Client Performance
• End-to-End Performance (Real Browser + Real System)

• Workload
  • generated workloads (dev/test/staging)
  • real-world user load (production)
• Monitoring/Reporting (stats, graphs)
  • real-time analytics
  • probing/tracing/profiling
  • aggregate reports from logs and tools
  • error reporting
  • multi-run comparisons

• Virtual User (VU) - lower level simulation
  • high number of VU per load generator
  • clientless web protocol simulation
  • more difficult to script
• Real Browser User (RBU) - drives real browsers
  • low number of RBU per load generator
  • selenium, grid/cloud
• Multi-Mechanize runs Virtual User (VU) scripts
  • no browser
  • client-side scripts/content not executed or rendered

Virtual User scripts run in parallel (with timers and assertions):

• generate user workload
• aggregate timing/results
• app/service is a black-box

"[Web] Performance and Load Testing Framework"

• Open Source (LGPLv3)
• Python 2 (2.6+)
• Web: http://multimechanize.com
• Project Hosted on Google Code (Git)
  • Download
  • Code Samples / Script Dev Guide
  • Wiki
  • Discussion Group

(probably moving to a dedicated web home soon)

"Python framework for performance and load testing remote APIs"

• Enables you to:
  • run simultaneous VU client scripts to generate workload against a remote service
  • run transactions at an increasing series of load levels against your API and measure performance
  • simulate realistic and projected API usage
  • collect/aggregate results for analysis (html reports with graphs, or xml)

You write the virtual user (client) scripts in Python, so any remote protocol/lib you want:

• HTTP (mechanize, urllib2, httplib, requests)
• Web Forms / Stateful programmatic browsing (mechanize)
• REST/SOAP/XMLRPC
• Sockets
• Database, NoSQL, Cache
• Any Python API

• Free (LGPLv3 licensed and $0 price)
• Runs on multiple platforms (nix, win)
• Python code base
• Scripts are in Python (batteries included, python libraries)
• Scalable/Distributable load generator
• Automatic report generation
• Aggregate stats (response times, throughput, percentiles)
• Graphs (scatter, time-series)

• Multi-process/Multi-threaded Test Harness
• Virtual User Scripting in Python
• Configurable Test Runner
• Report Generator (matplotlib)
• Input:

• Virtual User Test Scripts
• Config File

• Output:

• HTML Report with PNG images (graphs)
• XML Report (JMeter JTL format)

Each test script must implement a Transaction() class. This class must implement a run() method.

During a test run, your Transaction() class is instantiated once, and then its run() method is called repeatedly:

class Transaction(object):

    def __init__(self):
        # this gets called once

    def run(self):
        # this gets called repeatedly

A full script that will issue an HTTP GET using urllib2:

import urllib2

class Transaction(object):
    def run(self):
        url = 'http://www.example.com/'
        urllib2.urlopen(url).read()

mechanize request with assertions:

import mechanize

class Transaction(object):
    def run(self):
        br = mechanize.Browser()
        br.set_handle_robots(False)
        resp = br.open('http://www.example.com/')
        resp.read()

        assert (resp.code == 200), 'Bad HTTP Response'
        assert ('Example Web Page' in resp.get_data()),
            'Failed Content Verification'

sample config file:

[global]
run_time: 600
rampup: 600
results_ts_interval: 60
console_logging: off
xml_report: on

[user_group-1]
threads: 25
script: view_all_new.py

[user_group-2]
threads: 25
script: view_all_top.py

...

• test summary
  • transaction timers
  • custom timers (from instrumented client code)
• time-series/interval data
  • counts
  • rate/throughput
  • response times
  • average, min, max, stdev
  • percentiles (80th, 90th, 95th)

Sample Report: http://tinyurl.com/mm-results

Jenkins-CI:

• forked from Hudson
• results integration
• results repository
• job scheduler
• build trigger
• plugins

Hudson Helper (Android App) on DroidX:

"Continuous Performance"

Jenkins-CI + Multi-Mechanize

• Multi-Mechanize invoked from Jenkins CI server
• VU scripts written in Python (SST)
• results reported back to Jenkins in JMeter JTL
• consumed by Jenkins Performance Plugin

• AUT: Reddit clone site
• "reader/viewer users" scripted (read-heavy)
• Jenkins running Multi-Mechanize
  • 2 min steady-state load benchmark
  • 10 min increasing load test
• Jenkins Performance Plugin results display
  • trending
  • individual runs
  • alert thresholds

• Thank You.
• Questions?

Corey Goldberg
email: corey@goldb.org
web: http://goldb.org

• Slides: http://tinyurl.com/bpwebperf
• Related Links:
  • Multi-Mechanize: http://multimechanize.com
  • Jenkins-CI: http://jenkins-ci.org