A data-preserving update is a separate procedure, which needs further exploration and definition in this runbook.
The OSAF Hosted Service will manage a "Preview" release in synchronization with the overall OSAF effort. The important tracking items are:
This effort is composed of these following "must have" tasks:
Additional Preview "nice-to-haves" tasks are:
Some inactive tasks for the service include:
These major milestones are important to track for the OSAF Hosted Service Preview launch.
2007-01-24: Place production server order
2007-01-30: Terms of service tenets and revised draft available
2007-02-04: osaf.us updated to Cosmo 0.6.0 (with data migration)
2007-02-05: Preview metrics list drafted
2007-02-15: osaf.us/ shows web presence (instead of Cosmo redirect)
2007-02-18: Final domain name secured
2007-02-20: Metrics infrastructure running regularly to produce updates
2007-02-20: Production hardware up at colo
2007-02-24: osaf.us also available via final domain name
2007-02-28: osaf.us updated to Cosmo 0.6.1 (with data migration)
2007-03-05: Metrics available on dashboard
2007-03-10: Public help email (for password changes, etc) available
2007-03-15: Final name, marks, domain, and URLs
2007-03-25: NOC runbook available
2007-03-25: Presence revised to incorporate final name and domain in copy and marks
2007-03-30: Web presence look-and-feel at sufficient quality, signed-off
2007-03-30: osaf.us updated to Cosmo 0.7.0 (with data migration)
To achieve the OSAF Hosted Service Preview launch, the following dependencies need to be met and resources made available:
Solution branding
Solution name
Hosted service full name
Logo graphical mark (vector)
Cosmo production releases
Chandler production releases
Domain name registration
Legal council for terms of service
Visual design and production graphics for web presence
Copy proofing
QA (specifics TBD)
KEI IT
Barracuda spam filtering
Ticket tracking system (RT) and dedicated queues
NOC (Network operations center)
Colocation space (6U)
Bandwidth (<1mbps)
Last update: 2006-01-19
Provide a simple, but complete web presence for service; a "wrapper" around Cosmo.
Currently, most URLs direct to Cosmo homepage; wrapper placeholder available at osaf.us/dashboard
Should have a consistent design and look-and-feel with other OSAF properties
Service should probably "sit inside" the same site (domain) as other products
2007-02-15: osaf.us/ shows web presence (instead of Cosmo redirect)
Wireframe
Match cosmo.o11n.org/chandler.o11n.org design using osaf.us
Signoff on final service naming
2007-02-18: Final domain name secured
2007-02-24: osaf.us also available via final domain name
2007-02-20: Wireframe of integrated site proposed
2007-03-01: Review of proposed marks
2007-03-15: Final name, marks, domain, and URLs
2007-03-25: Presence revised to incorporate final name and domain in copy and marks
2007-03-30: Web presence look-and-feel at sufficient quality, signed-off
Registration of domain names
Redo current site with chandler/cosmo.o11n.org look-and-feel
Acquire/procure primary domain names
Make any secondary additional domain names/redirects operational
Finalize name and marks for hosted service
Service full name: Chandler Foo
Vector of full service logo
Bitmaps at useful sizes
Migrate new domain into simultaneous usage as osaf.us
Acquire and configure the domain name used for the Preview hosted service
Prepare backend and maintenance mechanism for "status of the service" page
Finalization of formal terms of service
Preparation of usage documentation, customer FAQs, etc
Good-enough art direction, graphical design, and copywriting for all osaf.us-specific web content
Polish to high-quality, commercial standards
Make XHTML compliant
Need to develop service/org-wide "sitemap"
Blocked, pretty much, until branding and domain final
Possible overlap/confusion with Cosmo design/look-and-feel
Need brand-wide URL design/discussions to determine final implementations
Need to revamp existing spec to attempt alignment with single-brand
Need to determine full name of service. Proposal include "Chandler Service" and "Chandler Online".
Service homepage on final domain name
Specifications "must have" (below) included
Should be capable of dynamic and session-based operations
URL namespace is documented, clean, and extensible
Backend in Python dynamic content technology
Probably just extend the existing Myghty implementation
Early draft specifications. Additional updates, review, and integration of these lists is pending.
Homepage
Terms of service. Privacy policy.
Email address for support
Getting additional help
Signup
Log in
Free signup and usage
Usage docs
List of clients supported
"How to configure client X" docs
Favicon (ICO format for IE6 support)
Template synced/identical with other landing areas
Short-domain URLs (www redirects to base domain) for main web presence
Separate "status of service" site (chandler.info)
Blog (with Atom feed)
Wiki
Forum
Ticket web UI
Copy: relationship to OSAF
Copy: relationship to Cosmo
Copy: relationship to Chandler desktop
Domain names
.com, .org, .net, .info, .mobi, .us, international
Trademark
Service mark
Copyright
Graphics elements
About
Signup
Help
Blog
Terms
Privacy
Help
Copyright
Contacting us
Service FAQs
Links to client usage docs
Links into OSAF wiki
Links to Chandler usage
Links to Cosmo usage
Links for Outlook, iCal, Evolution, Sunbird/Lightning
"How to set up clients" documentation
Outlook 2007 (Webcal) configuration
Outlook 2003 (TBD) configuration
iCal 2.x (Webcal) configuration
Sunbird/Lightning (CalDAV) configuration
Evolution (CalDAV) configuration
404 page
500 page
"All other errors" page
http://twitter.com/
http://www.icalshare.com/faq.php
Server
Production server
Staging server
Metrics
Dashboard
Cosmo
Interop
Outlook 2003/ics
Outlook 2007/webcal
iCal 2.x/webcal
iCal 3.x/CalDAV
Evolution/CalDAV
Email clients
SMS
Last update: 2006-12-31
Update osaf.us production service to Cosmo 0.6 release
Currently awaiting release
Estimated release date Jan 30, 2007
Need to run a data migration
Now until release: test
2007-02-02: Cosmo 0.6.0 is released
2007-02-04: osaf.us updated to Cosmo 0.6.0 (with data migration)
2007-02-26: Cosmo 0.6.1 is released
2007-02-28: osaf.us updated to Cosmo 0.6.1 (with data migration)
2007-03-28: Cosmo 0.7.0 is released
2007-03-30: osaf.us updated to Cosmo 0.7.0 (with data migration)
Test migration
Run migration
Watch for errors
Production data migration
URL changes
Wide acceptance after migration that new system is working
Installation of release candidate, with full migrated data
Manual test against release candidate
Cosmo 0.6 final release
Announce upcoming release and downtime
everyone@osafoundation.org, general@osafoundation.org
service-users@osaf.us
Specific 2 hour window announced 3 days ahead of time
Build new Tomcat and Cosmo instance: dogfood-03 on host app-dogfood
Service manager signoff to begin downtime
Switch on "service is down" mode: no changes
Run migration
Scripted procedure:
lab: /home/osaf.us/bin/manage stop,rebuild,configure mig06-test-post
lab: time /home/osaf.us/bin/mig-test
lab: /home/osaf.us/bin/manage start mig06-test-post
Down production instance: dogfood-02
Take final backup of production database: cosmo_dogfood_02
Take copy of final backup to use as import file
Update import file to use InnoDB for all tables
Update import file to use UTF-8 for all tables
Drop/create database cosmo_dogfood_03
Import the import file into cosmo_dogfood_03
Add three needed foreign key constraints to production tables
Run Cosmo 0.5-0.6 migration application on cosmo_dogfood_03
Take a copy of post-migrated DML-only, no schema structure
Drop/create cosmo_dogfood_03 database
Start dogfood_03 instance to recreate fresh cosmo_dogfood_03 schema
Stop dogfood_03 instance
Drop all rows from all cosmo_dogfood_03 tables
Import DML-only copy of migrated data
Start dogfood_03 instance
Turn on "service is up" mode
Run spot-check
Homepage
Root login
User login
User calendar
Runtime unit test
Announce on IRC
Collect feedback for 20 minutes of OSAF staff checks
Announce on service-dev
Announce on blog ==== Changes
Support casual collaborator workflows
Include logging features
New signups now require round-trip email validation
Last update: 2007-01-14
Move OSAF Hosted Service to a set of servers capable of supporting a large number of users, specifically targeted for 10k regular Chandler users.
Should be in place in and well tested perhaps a month before Preview launch
2007-01-24: Place production server order
2007-02-20: Production hardware up at colo
Specify servers
Get quote
Place order
Receive order
Install OS
Configure OS
Install into colocation
Deprecate/shut-off cosmo-demo
1x server, 8 cores, 500Gb disk space (more is better)
CPU 2x Quad-core Intel Xeon LGA771 1333FSB 8MB
RAM 8x 4Gb FB PC5300 667Mhz ECC DDR2
Disk 8x 300Gb 15K SAS 3.5" hard drives
Hardware SAS RAID 8x card with battery backup
Redundant (N+1) power supply, hot-swap
Dual gig-E NICs
DVD-R slim drive
Rack sliding rails
3 year warranty
To be run by KEI IT
Purchase date: 2007-01-20
Probable vendor: ASA Computers
Probable CPU: Xeon Clovertown 5355 2.66Ghz
Possible chassis: SuperServer 6025B-3RB 2U
Possible RAID: Adeptec 4805SAS
Probable drives: Seagate Cheetah 15K.5 300Gb 3.5" ST3300655SS 5 Year Warranty
Debian Etch 64-bit
RAID 1+0 (3-disk RAID 0 sets, mirrored) + 2 hot spares
Partitioning 1Gb boot, 50Gb root, 50Gb spare, rest LVM physical
/home on 100Gb LV
/var on 300Gb LV
Accounts for Jared, Dave
Drive bays numbered and checked
JRockit R27.0 in /opt
Runit
Tomcat+Cosmo in /home/cosmos/
/home/osaf.us working area
Apache 2.2 with reverse proxy as runit service
Cosmo 0.6.0 instance as runit service
mysql-server-5.0 package
Vendor RAID management package (as runit service)
NTP synchronization via chrony
Versions of OS, architecture are correct
Available via IP KVM
Managed power is available
Dual power supplies are installed
Labeled with hostname and contact
Boots from scratch in final position, no-hands
Timezone, hwclock, localtime, and NTP are all working after final boot, no-hands
We don't really know if the application server or the database server portions of this configuration will "max out" first on this box. We have the ability, on pre-clustered versions of Cosmo, to separate the database onto a separate machine, yielding a 2-box cluster. Beyond that, we will (likely) need development support to properly support a very large user population.
Last update: 2007-01-25
Address core issues needed to collect base metrics
Technically, it's about more than the hosted service metrics; the overall community metrics are important to track and integrate too.
There's a few existing codebases which will likely be updated, integrated, and/or rewritten.
Metrics-related features can be rolled out incrementally.
Effort required pretty much scales linearly with the number of metrics and reports implemented (once a base infrastructure for analysis is complete).
2007-02-05: Preview metrics list drafted
2007-02-15: osaf.us has at least two metrics being regularly submitted to time-series web service
2007-02-20: Metrics infrastructure running regularly to produce updates
2007-03-05: Metrics available on dashboard
Launch+20d: Improved dashboard
Now
2h: Create metrics VM
Define "core management" metrics (list of 5)
Later
8h: Update the prototype time-series web service to CherryPy 3.0
Define full list of preview metrics available
15h: Build lightweight, dynamic, Python dashboard framework
8h: Write prototype code to parse Cosmo 0.6 access log
This work is a little behind; we should already have a shortlist of needed metrics agreed upon.
It's a little hard to specify properly until one really sees the data produced by new-logging-style Cosmo 0.6 in production.
More work is necessary to distinquish "metrics" (single numbers) from "reports" (tables/drill-down)
Are we ok with a fully-public metrics system?
Running regularly
Some counts of daily visitors broken down by user profile
A dashboard page is available to see the resulting analysis
(Strawman) list of "core management" metrics: Total number of unique visitors this week (broken down by category) Number of signups this week Number of Chandler downloads Retention analysis (users still using after 30/60/90/180/360 days of signup)
Dashboard has two base features: a graph of the "recent" history of the monitored metrics, and an ability to retrieve the history in a CSV format.
(Strawman) list of "preview" metrics: - People chandler users per week casual collaborators per week consultative users per week standalone users per week interop users per week number of signups per day percentage accesses by various clients - Traffic total hits per day total bandwidth (in/out) per day - Community number of Chandler downloads per week
Metrics are built on top of each other. From raw information, some "base" metrics are calculated. By combining (addition, subtraction, multiplication division, statistics, time analysis) the base metrics, additional metrics ("derived metrics") are calculated. All metrics carry units, which carry all the semantics and context of each metrics.
The hosted service metrics framework consists of three main components: Collection scripts Time-series web service * Dashboard
Most metrics are based on a time-series, which means measurements taken over time. Time-series metrics have handy properties, like being easy to visualize (ie, turn into graphs). They are useful for spotting trends that are important to product and strategic management.
We are very interested in "how many people are using our services", this lets us measure our impact on the public and guides our potential for sustainability. The most interesting metrics will therefore revolve around usage, people, frequency, and user profiles.
If the guiding principle is to "count people", propose core metric is "number of visitors per week", broken down into major user profiles:
chandler users per week
casual collaborators per week
consultative users per week
standalone users per week
interop users per week
So primary tenets for Preview metrics are:
Counting number of people using the system, matching user profiles, is a requirement
Community metrics should be revamped before Preview
Retention rates are very important to overall management, but complicated to implement. A "number of people using this week divided by total number of accounts signed up" will need to be included to serve as a primitive retention metric.
Two critical scaling factors are: 1) rate of updates, and 2) rate of synchronization. Rate of synchronization is hard, and we know what the default is for Chandler at least. But "average number of changes per day" should be included in Preview metrics.
Additional dimensions:
Change of events vs change of other stamps
Read vs writes
Authenticated versus anonymous
Recency, frequency, depth
Bandwidth
Performance
Low hanging fruit:
total hits per day
total bandwidth
total accounts accessed per day
chandler users per day
anonymous hits per day
percentage breakdown of chandler users by OS/platform
percentage breakdown of chandler user by version ID
percentage breakdown of all user agents
percentage breakdown by top-level domain
percentage of users from domains "probably" outside US
number of emails submitted to support queue
number of signups per day
number of authenticated logins per day
average number of items per collection
average number of collections per user
Important, non-trivial:
retention
consultative, standalone, mashup usage percentages
depth, average session length/ops
number of updates via chandler per user
getting user counts vs hit counts (interop users per day kinda hard)
weekly views of all "per day" metrics, in order to smooth daily fluctation, and better match spread-out usage patterns.
number of casual collaborators per chandler hub user
Need platform support:
Add authenticated user to Tomcat access_log
Clear marker for successful logins
Clear marker for anonymous access to collection
Clear marker for Chandler update
Clear marker for via-web-UI update
Clear marker for names of people using Cosmo UI
Implementations:
chandler visitors per day: distinct accounts accessed with authenticated access users with chandler user-agent
casual collaborators per day: "entry" urls followed using anonymous authentication to shared collection
consultative users per day: distinct accounts accessed with authenticated access with cosmo ui who have accessed with a chandler-user agent in the last 7 days
standalone users per day: distinct accounts accessed with authenticated access with non-chandler user-agent who have not accessed with a chandler-user agent in the last 7 days
interop users per day: distinct authenticated accounts acccess with non-chandler user agents, or maybe all accesses to non-cosmo UI URLs
retention: number of users who have had an access with 30 days of signup, 60 days, 90d, 180d, 1y
Last update: 2006-12-30
Provide a clear help and support system for service users
Currently have internally-published addresses
OSAF will not build service capacity ahead of the demand for it.
Support labor costs are a large component of operational costs and should be kept low; self-service is assumed to major component of how we'll manage the customer support load.
2007-02-01: Internal-review draft available
2007-03-10: Public help email (for password changes, etc) available
Preview: TOS published on web site
Have internal review of last-state
Identify legal representation
Reviewed by lawyer
Posted on site
Easy-to-find on site
Publish help@osaf.us on the service homepage
help@osaf.us goes through KEI Barracuda and then into KEI RequestTracker instance and a service-specific queue. Submissions initially emailed to service-dev as well for audit and awareness.
Last update: 2006-01-19
Provide service customers a clear, published terms of service and privacy policy.
Currently awaiting re-kickoff.
A substantial portion of work was done by pro-bono lawyer 12 months ago. Should use that as a base.
Terms of service must be in line with OSAF values.
The service will be used by both authenticated/personal-account access, as well anonymous users (say via tickets). Terms need to be broad.
2007-01-19: Legacy draft document distributed internally
2007-01-30: Terms of service tenets and revised draft available
2007-02-05: Primary lawyer resource identified
2007-02-25: Pretty-darn-solid draft available
2007-03-25: Post final TOS document
Now
Define tenets
Review existing draft
Identify Preview lawyer who will review/update/sign-off on TOS
Later
Integrate comments into new draft
Finalize draft
Post terms of service
Need to strike nice balance of protections and tone
Need to identify lawyer resource
Need to pick a name to refer to the service by in the TOS. First draft says "OSAF Service". Current best proposal is "Chandler Service".
What constitutes acceptance of the terms? Implicit in any anonymous or authenticated use?
What belongs in a TOS vs a privacy policy?
Any additional
If a user "owns their data", and you lose or break their data, need to carefully disclaim liability or assume risk.
Would like to maintain "common carrier" status and avoid assuming publisher risk.
Subject to any handicap-accessibility regulations?
Pre-set usage maximums or just say "reasonable"?
Protects OSAF
Protects customers
Reviewed by lawyer
Posted on site
Easy-to-find on site
Pick up last TOS developed for our service. Distribute around. Fix up and integrate comments. Have reviewed by one more lawyer. Signoff internally. Post publicly.
We'll review a few other terms of service for ideas: - http://twitter.com/tos - http://code.google.com/tos.html
Some work Jared did inititally (2005) to define tenets, needed terms, and other specifications can be dug up someday. We'll probably start with a new, community list.
Plain language
You own your data. We will provide mechanisms for you to retrieve the contents of any records or data you store in our service.
You can't sue us if we lose or corrupt your data, or if you can't get to your data.
We will not sell, rent, or otherwise make available your account information, data, or transaction history to anyone. Except where required to be handed over by law.
We can revise these policies.
You won't do things that endanger the viability of the service for others. This can mean "too much" usage, illegal usage, attacks on other accounts, etc.
Standard terms: Jurisdiction, Severability, Limitations of liability to fees paid, No Warranty, Indemnification, Entire Agreement, Internet not under our control, Assignment,
No fees for usage
We need to be able to monitor the system, possibly inspect your data during debugging or service investigations.
We comply with DCMA. We'll try to contact you, and will protect your rights, but your data may be taken down.
A clear, well-posted, user-friendly terms of service is an strong consumer expectation for any hosted service. OSAF should satisfy this requirement and provide a good terms of service.
Last update: 2007-01-08
Provide an easy-to-track, lightweight channel channel for people to track changes to the service.
Not a must-have for Preview, but having something would be nice.
Easy to do if it starts as just an "announce" channel, so as to skip comment-related effort.
Target 2007-01-12: Have all comments checked, with simple HTML page having all announcements
Target 2007-02-15: Have home-page "last 3" announcements in-line with a more link
Target 2007-03-01: Single-page HTML and RSS feeds in production
Now
Implement a read-only blog
Seed blog with historical postings
Later
Update to a best-practices blog
Users can subscribe via RSS to a list of service announcements
Collect all posts into individual files (format probably XHTML snippet). Name them so they sort lexically in a directory. Update the Myghty web site wrapper to process them and put them on a page, both HTML and ATOM.
Order: 1) single-page with all; 2) home-page blurb; 3) RSS feeds; 4) comments
Support service-to-consumer communications. In particular, broad announcement channels help to manage rapid changes possible in early-phase services.
Last update: 2007-01-08
Provide product management, business development, and promotional insight by showing core metric trends visually on a web page.
Business aphorism: you can't manage what you can't measure.
The dashbaord provides the primary presentation of metrics maintained for the hosted service. The dashboard selects a few "core metrics" and presents them well and in context with other metrics.
Of the hundreds of possible metrics, it's inevitable that management attention and focus become centered on a smaller shortlist of core metrics. These receive priority attention in the dashboard.
Need a backend implementation of core metrics before presentation.
Traditionally, we're talking graphs or tables kept up-to-date on an HTML page.
There's a basic daily and weekly analysis codebase available that will likely be reused.
Open to the public or password-protected only? Would love to just do it all public and retract some later if needed.
Need a graphing engine.
Inclined to keep around all source data and generate graphs as-needed. Avoid RRD implementation (round-robin) data backend for now (and thus probably avoid RRD graphs).
Target 2007-02-15: Firstgraphs on dashboard page
2007-03-25: NOC runbook available
To document systems operations procedures sufficiently to provide solid guidance to network operations staff.
To upgrade to a new production instance of Cosmo on osaf.us:
Edit /home/osaf.us/conf/cosmo-service-instance.conf. Find the old production configuration stanza, and mostly cut-and-paste it into a new instance definition. You will need to decide:
instance root (/home/cosmos/INSTANCE is the standard)
Cosmo and Snarf SVN URLs and revision numbers
MySQL database name
Tomcat HTTP port number (and shutdown port number)
Reverse proxy configuration Save the configuration file.
Create the new instance: sudo -u cosmo /home/osaf.us/bin/manage build,start INSTANCE_NAME
Check in changes into osaf.us svn repository.
A data-preserving update is a separate procedure, which needs further exploration and definition in this runbook.
To update an instance's configuration:
Edit /home/osaf.us/conf/cosmo-service-instance.conf. Tweak what you need, probably a port number or something.
Run: sudo -u cosmo /home/osaf.us/bin/manage stop,config,start INSTANCE_NAME
You can also reconfigure multiple instances in one command by placing additional instance names on the bin/manage command line, spearated by spaces. This is useful for taking one instance off the proxy_port config, and putting another one into that config quickly.
The production hardware:
Production Cosmo+Postgres server
ASA Computers
1x SuperServer 6025B-3RB 2U
2x Xeon Clovertown 5355 2.66Ghz
8x Seagate Cheetah 15K.5 300Gb 3.5" ST3300655SS 5 Year Warranty
1x Adeptec 4805SAS, battery backup
Debian Etch 64-bit
RAID 1+0 (3-disk RAID 0 sets, mirrored) + 2 hot spares
Partitioning 1Gb boot, 50Gb root, 50Gb spare, rest LVM physical
/home on 100Gb LV
/var on 300Gb LV
JRockit R27.0 in /opt
Runit
Tomcat+Cosmo in /home/cosmos/
/home/osaf.us working area
Apache 2.2 with reverse proxy as runit service
Should get user-friendly error pages when Cosmo is down and with
Apache keeps three files. HTTP access (extended common log format), error log (Apache logging), and Apache detailed format. Uses cronolog for rotation.
Tomcat keeps access log (common log format), cosmo log (osafsrv.log).
Nagios configuration (provides NOC-level availability alerts) - Production service HTTP:80 is up - "Runtime integration test" script succeeds (signup, put, get, report, webcal, delete)