Overview
There are lots of frameworks that works on appengine, so people might wonder which framework is the best one for them. So, I did a simple benchmark to see how much cpu time to warm-up various frameworks for appengine with minimized applications.
How to test
The rule is quite simple.- Use template system for rendering
- Use very simple template
- No middleware, context_processors
- The view passes single string ('hello') to the template
- No i18n (USE_I18N=False)
- webapp+django template
- Kay framework repository tip (changeset: 6c871008872a)
- app-engine-patch-1.1RC
- Upload an application
- Access by browser once
- View logs on admin console and record ms and cpu_ms values
- Go to next framework in turn and do the same test
Results
And I did this series of tests five times and got following result:Cold Starting result
webapp+django template
10-20 09:29AM 32.796 / 200 332ms 369cpu_ms 10-20 09:31AM 45.991 / 200 278ms 330cpu_ms 10-20 09:33AM 56.086 / 200 352ms 369cpu_ms 10-20 09:36AM 16.630 / 200 245ms 350cpu_ms 10-20 09:39AM 13.148 / 200 266ms 350cpu_ms
Kay
10-20 09:30AM 13.555 / 200 504ms 700cpu_ms 10-20 09:32AM 27.076 / 200 436ms 641cpu_ms 10-20 09:34AM 37.841 / 200 417ms 621cpu_ms 10-20 09:37AM 01.469 / 200 471ms 641cpu_ms 10-20 09:41AM 40.874 / 200 454ms 660cpu_ms
app-engine-patch
10-20 09:31AM 00.640 / 200 663ms 1010cpu_ms 10-20 09:33AM 12.117 / 200 594ms 991cpu_ms 10-20 09:35AM 29.921 / 200 654ms 1030cpu_ms 10-20 09:38AM 00.888 / 200 629ms 1030cpu_ms 10-20 09:46AM 07.109 / 200 702ms 1108cpu_msAdditionally, I did some tests for serving by hot instances:
Served by hot instances result
webapp+django template
10-20 09:39AM 23.025 / 200 6ms 3cpu_ms 10-20 09:39AM 24.104 / 200 6ms 5cpu_ms 10-20 09:39AM 25.212 / 200 9ms 5cpu_ms 10-20 09:39AM 26.310 / 200 6ms 4cpu_ms 10-20 09:39AM 27.414 / 200 7ms 4cpu_ms
Kay
10-20 09:41AM 42.991 / 200 13ms 5cpu_ms 10-20 09:41AM 45.691 / 200 6ms 5cpu_ms 10-20 09:41AM 46.538 / 200 7ms 5cpu_ms 10-20 09:41AM 47.506 / 200 9ms 8cpu_ms 10-20 09:41AM 48.536 / 200 8ms 5cpu_ms
app-engine-patch
10-20 09:46AM 09.021 / 200 10ms 10cpu_ms 10-20 09:46AM 09.949 / 200 8ms 8cpu_ms 10-20 09:46AM 10.869 / 200 8ms 6cpu_ms 10-20 09:46AM 11.834 / 200 11ms 11cpu_ms 10-20 09:46AM 12.743 / 200 9ms 10cpu_ms
The application used in these tests
The template used is common among these frameworks.
index.html:<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd"%gt;
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<title>Top Page - myapp</title>
</head>
<body>
{{ message }}
</body>
</html>
Here are the application settings for each frameworks: webapp+django template
main.py:#!/usr/bin/env python
#
# Copyright 2007 Google Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
import logging
import os
import wsgiref.handlers
from google.appengine.ext import webapp
from google.appengine.ext import db
from google.appengine.ext.webapp import template
class MainHandler(webapp.RequestHandler):
def get(self):
contents = {'message': 'hello'}
path = os.path.join(os.path.dirname(__file__), 'index.html')
self.response.out.write(template.render(path, contents))
def main():
logging.getLogger().setLevel(logging.debug)
application = webapp.WSGIApplication([('/', MainHandler)],
debug=True)
wsgiref.handlers.CGIHandler().run(application)
if __name__ == '__main__':
main()
Kay
settings.py:# -*- coding: utf-8 -*-
"""
A sample of kay settings.
:Copyright: (c) 2009 Accense Technology, Inc.
Takashi Matsuo ,
All rights reserved.
:license: BSD, see LICENSE for more details.
"""
DEFAULT_TIMEZONE = 'Asia/Tokyo'
DEBUG = False
SECRET_KEY = 'ReplaceItWithSecretString'
SESSION_PREFIX = 'gaesess:'
COOKIE_AGE = 1209600 # 2 weeks
COOKIE_NAME = 'KAY_SESSION'
ADD_APP_PREFIX_TO_KIND = True
ADMINS = (
)
TEMPLATE_DIRS = (
)
USE_I18N = False
DEFAULT_LANG = 'en'
INSTALLED_APPS = (
'myapp',
)
APP_MOUNT_POINTS = {
'myapp': '/',
}
CONTEXT_PROCESSORS = (
)
JINJA2_FILTERS = {
}
MIDDLEWARE_CLASSES = (
)
AUTH_USER_BACKEND = 'kay.auth.backend.GoogleBackend'
AUTH_USER_MODEL = 'kay.auth.models.GoogleUser'
myapp/urls.py: # -*- coding: utf-8 -*-
# myapp.urls
from werkzeug.routing import (
Map, Rule, Submount,
EndpointPrefix, RuleTemplate,
)
import myapp.views
def make_rules():
return [
EndpointPrefix('myapp/', [
Rule('/', endpoint='index'),
]),
]
all_views = {
'myapp/index': myapp.views.index,
}
myapp/views.py: # -*- coding: utf-8 -*-
# myapp.views
from kay.utils import render_to_response
# Create your views here.
def index(request):
return render_to_response('myapp/index.html', {'message': 'Hello'})
app-engine-patch
settings.py:# -*- coding: utf-8 -*-
from ragendja.settings_pre import *
# Increase this when you update your media on the production site, so users
# don't have to refresh their cache. By setting this your MEDIA_URL
# automatically becomes /media/MEDIA_VERSION/
MEDIA_VERSION = 1
# By hosting media on a different domain we can get a speedup (more parallel
# browser connections).
#if on_production_server or not have_appserver:
# MEDIA_URL = 'http://media.mydomain.com/media/%d/'
# Add base media (jquery can be easily added via INSTALLED_APPS)
COMBINE_MEDIA = {
'combined-%(LANGUAGE_CODE)s.js': (
# See documentation why site_data can be useful:
# http://code.google.com/p/app-engine-patch/wiki/MediaGenerator
'.site_data.js',
),
'combined-%(LANGUAGE_DIR)s.css': (
'global/look.css',
),
}
# Change your email settings
if on_production_server:
DEFAULT_FROM_EMAIL = 'bla@bla.com'
SERVER_EMAIL = DEFAULT_FROM_EMAIL
# Make this unique, and don't share it with anybody.
SECRET_KEY = '1234567890'
#ENABLE_PROFILER = True
#ONLY_FORCED_PROFILE = True
#PROFILE_PERCENTAGE = 25
#SORT_PROFILE_RESULTS_BY = 'cumulative' # default is 'time'
# Profile only datastore calls
#PROFILE_PATTERN = 'ext.db..+\((?:get|get_by_key_name|fetch|count|put)\)'
# Enable I18N and set default language to 'en'
USE_I18N = False
LANGUAGE_CODE = 'en'
# Restrict supported languages (and JS media generation)
LANGUAGES = (
('de', 'German'),
('en', 'English'),
)
TEMPLATE_CONTEXT_PROCESSORS = (
)
MIDDLEWARE_CLASSES = (
)
# Google authentication
AUTH_USER_MODULE = 'ragendja.auth.google_models'
AUTH_ADMIN_MODULE = 'ragendja.auth.google_admin'
# Hybrid Django/Google authentication
#AUTH_USER_MODULE = 'ragendja.auth.hybrid_models'
LOGIN_URL = '/account/login/'
LOGOUT_URL = '/account/logout/'
LOGIN_REDIRECT_URL = '/'
INSTALLED_APPS = (
# Add jquery support (app is in "common" folder). This automatically
# adds jquery to your COMBINE_MEDIA['combined-%(LANGUAGE_CODE)s.js']
# Note: the order of your INSTALLED_APPS specifies the order in which
# your app-specific media files get combined, so jquery should normally
# come first.
'appenginepatcher',
'ragendja',
'myapp',
)
# List apps which should be left out from app settings and urlsauto loading
IGNORE_APP_SETTINGS = IGNORE_APP_URLSAUTO = (
# Example:
# 'django.contrib.admin',
# 'django.contrib.auth',
# 'yetanotherapp',
)
# Remote access to production server (e.g., via manage.py shell --remote)
DATABASE_OPTIONS = {
# Override remoteapi handler's path (default: '/remote_api').
# This is a good idea, so you make it not too easy for hackers. ;)
# Don't forget to also update your app.yaml!
#'remote_url': '/remote-secret-url',
# !!!Normally, the following settings should not be used!!!
# Always use remoteapi (no need to add manage.py --remote option)
#'use_remote': True,
# Change appid for remote connection (by default it's the same as in
# your app.yaml)
#'remote_id': 'otherappid',
# Change domain (default: .appspot.com)
#'remote_host': 'bla.com',
}
from ragendja.settings_post import *
urls.py(minimized): # -*- coding: utf-8 -*-
from django.conf.urls.defaults import *
from ragendja.urlsauto import urlpatterns
from ragendja.auth.urls import urlpatterns as auth_patterns
urlpatterns = auth_patterns + patterns('',
(r'', include('myapp.urls')),
) + urlpatterns
myapp/urls.py: # -*- coding: utf-8 -*- from django.conf.urls.defaults import * urlpatterns = patterns( 'myapp.views', url(r'^$', 'index', name='myapp_index'), )myapp/views.py:
# -*- coding: utf-8 -*-
from ragendja.template import render_to_response
def index(request):
return render_to_response(request, 'myapp/index.html',
{'message': 'Hello'})
Conclusion
There are significant defferences among warm-up costs of these three frameworks. But when it comes to serving by warm instances, the defferences are rather small.
Actually you can choose whatever you want, but what if you must to choose one framework among these three options?
webapp+django template is really fast(actually without django template, webapp is much faster than this, but I think its unfair ;-P), so If you really need speed, webapp might be the best option.
Having said that, sometimes we need a fancy way for constructing rather complex applications. Perhaps you can choose Kay/app-engine-patch if you can accept 2 or 3 times(obviously compared with webapp) costs on cold-startup.
If you love Django admin capability and hardly throw it away, only app-engine-patch could be your option.
Actually I don't know much about app-engine-patch, so these settings is not enough. Please let me know if that's the case. If you make some tests similar to these tests, I'd be glad to know the result.
The result with tipfy is added.
tipfy is one of the fastest and lightest frameworks. You can use a very nice debugger(werkzeug's) in dev environment(This debugger is also available with Kay).
Cold Starting result
tipfy
10-20 03:43PM 28.814 / 200 411ms 602cpu_ms 10-20 03:44PM 31.751 / 200 367ms 563cpu_ms 10-20 03:45PM 09.913 / 200 407ms 563cpu_ms 10-20 03:46PM 23.932 / 200 413ms 602cpu_ms 10-20 03:47PM 19.530 / 200 416ms 583cpu_ms
Served by hot instances result
tipfy
10-20 03:47PM 21.859 / 200 6ms 4cpu_ms 10-20 03:47PM 23.003 / 200 8ms 5cpu_ms 10-20 03:47PM 24.177 / 200 6ms 4cpu_ms 10-20 03:47PM 25.166 / 200 6ms 4cpu_ms 10-20 03:47PM 26.181 / 200 5ms 4cpu_ms
Application
tipfy
urls.py:# -*- coding: utf-8 -*-
"""
urls
~~~~
URL definitions.
:copyright: 2009 by tipfy.org.
:license: BSD, see LICENSE.txt for more details.
"""
from tipfy import Rule
urls = [
Rule('/', endpoint='home', handler='hello:HelloWorldHandler'),
]
hello.py: from tipfy import RequestHandler
from tipfy.ext.jinja2 import render_response
class HelloWorldHandler(RequestHandler):
def get(self, **kwargs):
context = {'message': 'hello'}
return render_response('index.html', **context)
15 comments:
Really useful to have these stats - thanks! It'd be nice to see more tests for more frameworks and for more variations on apps - it would really help when it comes to pointing out specific options for people who are seeing multi-second startup times.
It would be interested to see how web.py fairs against those also. I suspect it could be even faster than webapp + django template, since web.py is very similar to webapp and its templates are pre-compliled.
how about pylons?
I think that we can improve some of kay's warmup costs by getting rid of some global imports. Some aren't really needed.
Arachnid: Thanks.
Anh: OK. I'll add web.py result if there's a time.
Bartosz: Actually I'm not so eager to test pylons. sorry.
Ian: Thanks for suggestion. Actually kay.i18n.__init__.py has unnecessary import of simplejson, so I've just removed it and got faster result!
Yah, I think some places import auth and some other things. We could move some imports inside of functions to make the modules load faster.
I'll take a look at it this weekend at Onsen.
Very interesting !
Could you add another bench with Tornado : http://www.tornadoweb.org/
Here is the code (didn't test but it should work)
http://pastebin.com/m10dbedda
The app-engine-patch example isn't 100% correct. In your view, please use django.shortcuts.render_to_response instead of the ragendja handler.
It would also be interesting to see test results with a more practical real-world project (i.e., authentication, context processors, etc.). Even though you don't use django.forms, they always get loaded by app-engine-patch (and Django would also load them when you import django.db.models).
Waldemar: Thanks for pointing out! I've just tried as you said, but there is no significant differences.
Have you looked at the template module from Tornado at all?
Thanks for posting these. I have been astonished at how long these cold start times are when starting to develop with Grails and also Gaelyk. If you have a chance to add benchmarks for these to your page I think it would be of value to the community.
Takashi, could you please do another measurement with our native Django port: http://bitbucket.org/wkornewald/django-nonrel-multidb/
Thanks for taking the time to compare the startup times for these frameworks!
Hi I liked your information and would like to continue posting more about it .... in fact I did a while ago to work in or call virtual laser keyboard on this subject and I was fascinated ... thanks
Post a Comment