Moon blue diary: python

Showing posts with label python. Show all posts

November 2, 2009

pkg_resources の warning を抑制する

sitecustomize.py を site-packages 内に下記の内容で作成する

import warnings
warnings.filterwarnings('ignore',
  message=r'Module .*? is being added to sys\.path', append=True)

ここに書いてありました:
http://lucumr.pocoo.org/2008/2/19/sick-of-pkg-resources-warnings

October 21, 2009

Minimum cost for warming-up various frameworks(and more)

Overview

There are lots of frameworks that works on appengine, so people might wonder which framework is the best one for them. So, I did a simple benchmark to see how much cpu time to warm-up various frameworks for appengine with minimized applications.

How to test

The rule is quite simple.

Use template system for rendering
Use very simple template
No middleware, context_processors
The view passes single string ('hello') to the template
No i18n (USE_I18N=False)

Today, I tested 3 frameworks;

webapp+django template
Kay framework repository tip (changeset: 6c871008872a)
app-engine-patch-1.1RC

The single series of tests was made as follows;

Upload an application
Access by browser once
View logs on admin console and record ms and cpu_ms values
Go to next framework in turn and do the same test

Results

And I did this series of tests five times and got following result:

Cold Starting result

webapp+django template

10-20 09:29AM 32.796 / 200 332ms 369cpu_ms
10-20 09:31AM 45.991 / 200 278ms 330cpu_ms
10-20 09:33AM 56.086 / 200 352ms 369cpu_ms
10-20 09:36AM 16.630 / 200 245ms 350cpu_ms
10-20 09:39AM 13.148 / 200 266ms 350cpu_ms

Kay

10-20 09:30AM 13.555 / 200 504ms 700cpu_ms
10-20 09:32AM 27.076 / 200 436ms 641cpu_ms
10-20 09:34AM 37.841 / 200 417ms 621cpu_ms
10-20 09:37AM 01.469 / 200 471ms 641cpu_ms
10-20 09:41AM 40.874 / 200 454ms 660cpu_ms

app-engine-patch

10-20 09:31AM 00.640 / 200 663ms 1010cpu_ms
10-20 09:33AM 12.117 / 200 594ms 991cpu_ms
10-20 09:35AM 29.921 / 200 654ms 1030cpu_ms
10-20 09:38AM 00.888 / 200 629ms 1030cpu_ms
10-20 09:46AM 07.109 / 200 702ms 1108cpu_ms

Additionally, I did some tests for serving by hot instances:

Served by hot instances result

webapp+django template

10-20 09:39AM 23.025 / 200 6ms 3cpu_ms
10-20 09:39AM 24.104 / 200 6ms 5cpu_ms
10-20 09:39AM 25.212 / 200 9ms 5cpu_ms
10-20 09:39AM 26.310 / 200 6ms 4cpu_ms
10-20 09:39AM 27.414 / 200 7ms 4cpu_ms

Kay

10-20 09:41AM 42.991 / 200 13ms 5cpu_ms
10-20 09:41AM 45.691 / 200 6ms 5cpu_ms
10-20 09:41AM 46.538 / 200 7ms 5cpu_ms
10-20 09:41AM 47.506 / 200 9ms 8cpu_ms
10-20 09:41AM 48.536 / 200 8ms 5cpu_ms

app-engine-patch

10-20 09:46AM 09.021 / 200 10ms 10cpu_ms
10-20 09:46AM 09.949 / 200 8ms 8cpu_ms
10-20 09:46AM 10.869 / 200 8ms 6cpu_ms
10-20 09:46AM 11.834 / 200 11ms 11cpu_ms
10-20 09:46AM 12.743 / 200 9ms 10cpu_ms

The application used in these tests

The template used is common among these frameworks.

index.html:

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd"%gt;
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<title>Top Page - myapp</title>
</head>
<body>
{{ message }}
</body>
</html>

Here are the application settings for each frameworks:

webapp+django template

main.py:

#!/usr/bin/env python
#
# Copyright 2007 Google Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#


import logging
import os

import wsgiref.handlers


from google.appengine.ext import webapp
from google.appengine.ext import db
from google.appengine.ext.webapp import template

class MainHandler(webapp.RequestHandler):

  def get(self):
    contents = {'message': 'hello'}
    path = os.path.join(os.path.dirname(__file__), 'index.html')
    self.response.out.write(template.render(path, contents))


def main():
  logging.getLogger().setLevel(logging.debug)
  application = webapp.WSGIApplication([('/', MainHandler)],
                                       debug=True)
  wsgiref.handlers.CGIHandler().run(application)


if __name__ == '__main__':
  main()

Kay

settings.py:

# -*- coding: utf-8 -*-

"""
A sample of kay settings.

:Copyright: (c) 2009 Accense Technology, Inc. 
                     Takashi Matsuo ,
                     All rights reserved.
:license: BSD, see LICENSE for more details.
"""

DEFAULT_TIMEZONE = 'Asia/Tokyo'
DEBUG = False
SECRET_KEY = 'ReplaceItWithSecretString'
SESSION_PREFIX = 'gaesess:'
COOKIE_AGE = 1209600 # 2 weeks
COOKIE_NAME = 'KAY_SESSION'

ADD_APP_PREFIX_TO_KIND = True

ADMINS = (
)

TEMPLATE_DIRS = (
)

USE_I18N = False
DEFAULT_LANG = 'en'

INSTALLED_APPS = (
  'myapp',
)

APP_MOUNT_POINTS = {
  'myapp': '/',
}

CONTEXT_PROCESSORS = (
)

JINJA2_FILTERS = {
}

MIDDLEWARE_CLASSES = (
)
AUTH_USER_BACKEND = 'kay.auth.backend.GoogleBackend'
AUTH_USER_MODEL = 'kay.auth.models.GoogleUser'

myapp/urls.py:

# -*- coding: utf-8 -*-
# myapp.urls


from werkzeug.routing import (
  Map, Rule, Submount,
  EndpointPrefix, RuleTemplate,
)
import myapp.views

def make_rules():
  return [
    EndpointPrefix('myapp/', [
      Rule('/', endpoint='index'),
    ]),
  ]

all_views = {
  'myapp/index': myapp.views.index,
}

myapp/views.py:

# -*- coding: utf-8 -*-
# myapp.views

from kay.utils import render_to_response

# Create your views here.

def index(request):
  return render_to_response('myapp/index.html', {'message': 'Hello'})

app-engine-patch

settings.py:

# -*- coding: utf-8 -*-
from ragendja.settings_pre import *

# Increase this when you update your media on the production site, so users
# don't have to refresh their cache. By setting this your MEDIA_URL
# automatically becomes /media/MEDIA_VERSION/
MEDIA_VERSION = 1

# By hosting media on a different domain we can get a speedup (more parallel
# browser connections).
#if on_production_server or not have_appserver:
#    MEDIA_URL = 'http://media.mydomain.com/media/%d/'

# Add base media (jquery can be easily added via INSTALLED_APPS)
COMBINE_MEDIA = {
    'combined-%(LANGUAGE_CODE)s.js': (
        # See documentation why site_data can be useful:
        # http://code.google.com/p/app-engine-patch/wiki/MediaGenerator
        '.site_data.js',
    ),
    'combined-%(LANGUAGE_DIR)s.css': (
        'global/look.css',
    ),
}

# Change your email settings
if on_production_server:
    DEFAULT_FROM_EMAIL = 'bla@bla.com'
    SERVER_EMAIL = DEFAULT_FROM_EMAIL

# Make this unique, and don't share it with anybody.
SECRET_KEY = '1234567890'

#ENABLE_PROFILER = True
#ONLY_FORCED_PROFILE = True
#PROFILE_PERCENTAGE = 25
#SORT_PROFILE_RESULTS_BY = 'cumulative' # default is 'time'
# Profile only datastore calls
#PROFILE_PATTERN = 'ext.db..+\((?:get|get_by_key_name|fetch|count|put)\)'

# Enable I18N and set default language to 'en'
USE_I18N = False
LANGUAGE_CODE = 'en'

# Restrict supported languages (and JS media generation)
LANGUAGES = (
    ('de', 'German'),
    ('en', 'English'),
)

TEMPLATE_CONTEXT_PROCESSORS = (
)

MIDDLEWARE_CLASSES = (
)

# Google authentication
AUTH_USER_MODULE = 'ragendja.auth.google_models'
AUTH_ADMIN_MODULE = 'ragendja.auth.google_admin'
# Hybrid Django/Google authentication
#AUTH_USER_MODULE = 'ragendja.auth.hybrid_models'

LOGIN_URL = '/account/login/'
LOGOUT_URL = '/account/logout/'
LOGIN_REDIRECT_URL = '/'

INSTALLED_APPS = (
    # Add jquery support (app is in "common" folder). This automatically
    # adds jquery to your COMBINE_MEDIA['combined-%(LANGUAGE_CODE)s.js']
    # Note: the order of your INSTALLED_APPS specifies the order in which
    # your app-specific media files get combined, so jquery should normally
    # come first.
    'appenginepatcher',
    'ragendja',
    'myapp',
)

# List apps which should be left out from app settings and urlsauto loading
IGNORE_APP_SETTINGS = IGNORE_APP_URLSAUTO = (
    # Example:
    # 'django.contrib.admin',
    # 'django.contrib.auth',
    # 'yetanotherapp',
)

# Remote access to production server (e.g., via manage.py shell --remote)
DATABASE_OPTIONS = {
    # Override remoteapi handler's path (default: '/remote_api').
    # This is a good idea, so you make it not too easy for hackers. ;)
    # Don't forget to also update your app.yaml!
    #'remote_url': '/remote-secret-url',

    # !!!Normally, the following settings should not be used!!!

    # Always use remoteapi (no need to add manage.py --remote option)
    #'use_remote': True,

    # Change appid for remote connection (by default it's the same as in
    # your app.yaml)
    #'remote_id': 'otherappid',

    # Change domain (default: .appspot.com)
    #'remote_host': 'bla.com',
}

from ragendja.settings_post import *

urls.py(minimized):

# -*- coding: utf-8 -*-
from django.conf.urls.defaults import *
from ragendja.urlsauto import urlpatterns
from ragendja.auth.urls import urlpatterns as auth_patterns

urlpatterns = auth_patterns + patterns('',
    (r'', include('myapp.urls')),
) + urlpatterns

myapp/urls.py:

# -*- coding: utf-8 -*-
from django.conf.urls.defaults import *

urlpatterns = patterns(
  'myapp.views',
  url(r'^$', 'index', name='myapp_index'),
)

myapp/views.py:

# -*- coding: utf-8 -*-
from ragendja.template import render_to_response

def index(request):
  return render_to_response(request, 'myapp/index.html',
                            {'message': 'Hello'})

Conclusion

There are significant defferences among warm-up costs of these three frameworks. But when it comes to serving by warm instances, the defferences are rather small.

Actually you can choose whatever you want, but what if you must to choose one framework among these three options?

webapp+django template is really fast(actually without django template, webapp is much faster than this, but I think its unfair ;-P), so If you really need speed, webapp might be the best option.

Having said that, sometimes we need a fancy way for constructing rather complex applications. Perhaps you can choose Kay/app-engine-patch if you can accept 2 or 3 times(obviously compared with webapp) costs on cold-startup.

If you love Django admin capability and hardly throw it away, only app-engine-patch could be your option.

Actually I don't know much about app-engine-patch, so these settings is not enough. Please let me know if that's the case. If you make some tests similar to these tests, I'd be glad to know the result.

The result with tipfy is added.

tipfy is one of the fastest and lightest frameworks. You can use a very nice debugger(werkzeug's) in dev environment(This debugger is also available with Kay).

Cold Starting result

tipfy

10-20 03:43PM 28.814 / 200 411ms 602cpu_ms
10-20 03:44PM 31.751 / 200 367ms 563cpu_ms
10-20 03:45PM 09.913 / 200 407ms 563cpu_ms
10-20 03:46PM 23.932 / 200 413ms 602cpu_ms
10-20 03:47PM 19.530 / 200 416ms 583cpu_ms

Served by hot instances result

tipfy

10-20 03:47PM 21.859 / 200 6ms 4cpu_ms
10-20 03:47PM 23.003 / 200 8ms 5cpu_ms
10-20 03:47PM 24.177 / 200 6ms 4cpu_ms
10-20 03:47PM 25.166 / 200 6ms 4cpu_ms
10-20 03:47PM 26.181 / 200 5ms 4cpu_ms

Application

tipfy

urls.py:

# -*- coding: utf-8 -*-
"""
    urls
    ~~~~

    URL definitions.

    :copyright: 2009 by tipfy.org.
    :license: BSD, see LICENSE.txt for more details.
"""
from tipfy import Rule

urls = [
    Rule('/', endpoint='home', handler='hello:HelloWorldHandler'),
]

hello.py:

from tipfy import RequestHandler
from tipfy.ext.jinja2 import render_response

class HelloWorldHandler(RequestHandler):
  def get(self, **kwargs):
    context = {'message': 'hello'}
    return render_response('index.html', **context)

August 31, 2009

Using pyjamas on Kay

I use ${PYJAMAS_HOME} and ${PROJECT_DIR} variable for explanation here. ${PYJAMAS_HOME} is a directory in which you installed pyjamas. ${PROJECT_DIR} is your project directory.

For opener, please begin with customizing ${PYJAMAS_HOME}/examples/jsonrpc. You need only JSONRPCExample.html in the `public` directory.


$ tree kay-pyjamas-sample
kay-pyjamas-sample
|-- JSONRPCExample.py
`-- public
    `-- JSONRPCExample.html

1 directory, 2 files

Edit your own JSONRPCExample.py as follows:


import pyjd # dummy in pyjs

from pyjamas.ui.RootPanel import RootPanel
from pyjamas.ui.TextArea import TextArea
from pyjamas.ui.Label import Label
from pyjamas.ui.Button import Button
from pyjamas.ui.HTML import HTML
from pyjamas.ui.VerticalPanel import VerticalPanel
from pyjamas.ui.HorizontalPanel import HorizontalPanel
from pyjamas.ui.ListBox import ListBox
from pyjamas.JSONService import JSONProxy

class JSONRPCExample:
    def onModuleLoad(self):
        self.TEXT_WAITING = "Waiting for response..."
        self.TEXT_ERROR = "Server Error"
        self.METHOD_ECHO = "Echo"
        self.METHOD_REVERSE = "Reverse"
        self.METHOD_UPPERCASE = "UPPERCASE"
        self.METHOD_LOWERCASE = "lowercase"
        self.methods = [self.METHOD_ECHO, self.METHOD_REVERSE, self.METHOD_UPPERCASE, self.METHOD_LOWERCASE]

        self.remote_py = EchoServicePython()

        self.status=Label()
        self.text_area = TextArea()
        self.text_area.setText("""{'Test'} [\"String\"]
\tTest Tab
Test Newline\n
after newline
""" + r"""Literal String:
{'Test'} [\"String\"]
""")
        self.text_area.setCharacterWidth(80)
        self.text_area.setVisibleLines(8)
        
        self.method_list = ListBox()
        self.method_list.setName("hello")
        self.method_list.setVisibleItemCount(1)
        for method in self.methods:
            self.method_list.addItem(method)
        self.method_list.setSelectedIndex(0)

        method_panel = HorizontalPanel()
        method_panel.add(HTML("Remote string method to call: "))
        method_panel.add(self.method_list)
        method_panel.setSpacing(8)

        self.button_py = Button("Send to Python Service", self)

        buttons = HorizontalPanel()
        buttons.add(self.button_py)
        buttons.setSpacing(8)
        
        info = """JSON-RPC Example

        This example demonstrates the calling of server services with
           JSON-RPC.
        

        Enter some text below, and press a button to send the text
           to an Echo service on your server. An echo service simply sends the exact same text back that it receives.
           
"""
        
        panel = VerticalPanel()
        panel.add(HTML(info))
        panel.add(self.text_area)
        panel.add(method_panel)
        panel.add(buttons)
        panel.add(self.status)
        
        RootPanel().add(panel)

    def onClick(self, sender):
        self.status.setText(self.TEXT_WAITING)
        method = self.methods[self.method_list.getSelectedIndex()]
        text = self.text_area.getText()
        id = -1
        if method == self.METHOD_ECHO:
            id = self.remote_py.echo(text, self)
        elif method == self.METHOD_REVERSE:
            id = self.remote_py.reverse(text, self)
        elif method == self.METHOD_UPPERCASE:
            id = self.remote_py.uppercase(text, self)
        elif method == self.METHOD_LOWERCASE:
            id = self.remote_py.lowercase(text, self)
        if id<0:
            self.status.setText(self.TEXT_ERROR)

    def onRemoteResponse(self, response, request_info):
        self.status.setText(response)

    def onRemoteError(self, code, message, request_info):
        self.status.setText("Server Error or Invalid Response: ERROR %d - %s" %
                            (code, message))


class EchoServicePython(JSONProxy):
    def __init__(self):
        JSONProxy.__init__(self, "/json", ["echo", "reverse", "uppercase", "lowercase"])

if __name__ == '__main__':
    # for pyjd, set up a web server and load the HTML from there:
    # this convinces the browser engine that the AJAX will be loaded
    # from the same URI base as the URL, it's all a bit messy...
    pyjd.setup("http://127.0.0.1/examples/jsonrpc/public/JSONRPCExample.html")
    app = JSONRPCExample()
    app.onModuleLoad()
    pyjd.run()

Then, cd into this directory and compile it. This time, I use ${PROJECT_DIR}/media for a target directory.


$ cd kay-pyjamas-sample
$ ${PYJAMAS_HOME}/bin/pyjsbuild -o ${PROJECT_DIR}/media JSONRPCExample.py

If succeed, you'll get ${PROJECT_DIR}/media/JSONRPCExample.html. Please access http://localhost:8080/media/JSONRPCExample.html and, you'll get forms for testing jsonrpc.

Let's write server side code. First, you need to edit urls.py in your application.

${PROJECT_DIR}/myapp/urls.py


# -*- coding: utf-8 -*-
# myapp.urls


from werkzeug.routing import (
  Map, Rule, Submount,
  EndpointPrefix, RuleTemplate,
)
import myapp.views

def make_rules():
  return [
    EndpointPrefix('myapp/', [
      Rule('/', endpoint='index'),
      Rule('/json', endpoint='json_rpc'),
    ]),
  ]

all_views = {
  'myapp/index': myapp.views.index,
  'myapp/json_rpc': myapp.views.json_rpc,
}

You also need to edit views.py.

${PROJECT_DIR}/myapp/views.py


# -*- coding: utf-8 -*-
# myapp.views
# ... (snip)

import simplejson

# ... (snip)

def json_uppercase(args):
  return [args[0].upper()]

def json_echo(args):
  return [args[0]]

def json_reverse(args):
  return [args[0][::-1]]

def json_lowercase(args):
  return [args[0].lower()]

def json_rpc(request):
  if request.method:
    args = simplejson.loads(request.data)
    method_name = 'json_%s' % args[u"method"]
    json_func = globals().get(method_name)
    json_params = args[u"params"]
    json_method_id = args[u"id"]
    result = json_func(json_params)
    args.pop(u"method")
    args["result"] = result[0]
    args["error"] = None
    return Response(simplejson.dumps(args), content_type="application/json")
  else:
    return Response('Error')

That's all. Please enjoy :-)

Kay で pyjamas を使う

日本語の情報もできたのでちょっと変わり種をやります。

Python 版 GWT である pyjamas を Kay と組み合わせてみました.. と言っても jsonrpc の簡単なサンプルを動かしただけですが。

Pyjamas をインストールした場所を ${PYJAMAS_HOME} と、プロジェクトのディレクトリを ${PROJECT_DIR} として進めましょう。

${PYJAMAS_HOME}/examples/jsonrpc をいじって使います。public 以下は JSONRPCExample.html だけ必要みたい。


$ tree kay-pyjamas-sample
kay-pyjamas-sample
|-- JSONRPCExample.py
`-- public
    `-- JSONRPCExample.html

1 directory, 2 files

JSONRPCExample.py を下記のように少しいじります。


import pyjd # dummy in pyjs

from pyjamas.ui.RootPanel import RootPanel
from pyjamas.ui.TextArea import TextArea
from pyjamas.ui.Label import Label
from pyjamas.ui.Button import Button
from pyjamas.ui.HTML import HTML
from pyjamas.ui.VerticalPanel import VerticalPanel
from pyjamas.ui.HorizontalPanel import HorizontalPanel
from pyjamas.ui.ListBox import ListBox
from pyjamas.JSONService import JSONProxy

class JSONRPCExample:
    def onModuleLoad(self):
        self.TEXT_WAITING = "Waiting for response..."
        self.TEXT_ERROR = "Server Error"
        self.METHOD_ECHO = "Echo"
        self.METHOD_REVERSE = "Reverse"
        self.METHOD_UPPERCASE = "UPPERCASE"
        self.METHOD_LOWERCASE = "lowercase"
        self.methods = [self.METHOD_ECHO, self.METHOD_REVERSE, self.METHOD_UPPERCASE, self.METHOD_LOWERCASE]

        self.remote_py = EchoServicePython()

        self.status=Label()
        self.text_area = TextArea()
        self.text_area.setText("""{'Test'} [\"String\"]
\tTest Tab
Test Newline\n
after newline
""" + r"""Literal String:
{'Test'} [\"String\"]
""")
        self.text_area.setCharacterWidth(80)
        self.text_area.setVisibleLines(8)
        
        self.method_list = ListBox()
        self.method_list.setName("hello")
        self.method_list.setVisibleItemCount(1)
        for method in self.methods:
            self.method_list.addItem(method)
        self.method_list.setSelectedIndex(0)

        method_panel = HorizontalPanel()
        method_panel.add(HTML("Remote string method to call: "))
        method_panel.add(self.method_list)
        method_panel.setSpacing(8)

        self.button_py = Button("Send to Python Service", self)

        buttons = HorizontalPanel()
        buttons.add(self.button_py)
        buttons.setSpacing(8)
        
        info = """JSON-RPC Example

        This example demonstrates the calling of server services with
           JSON-RPC.
        

        Enter some text below, and press a button to send the text
           to an Echo service on your server. An echo service simply sends the exact same text back that it receives.
           
"""
        
        panel = VerticalPanel()
        panel.add(HTML(info))
        panel.add(self.text_area)
        panel.add(method_panel)
        panel.add(buttons)
        panel.add(self.status)
        
        RootPanel().add(panel)

    def onClick(self, sender):
        self.status.setText(self.TEXT_WAITING)
        method = self.methods[self.method_list.getSelectedIndex()]
        text = self.text_area.getText()
        id = -1
        if method == self.METHOD_ECHO:
            id = self.remote_py.echo(text, self)
        elif method == self.METHOD_REVERSE:
            id = self.remote_py.reverse(text, self)
        elif method == self.METHOD_UPPERCASE:
            id = self.remote_py.uppercase(text, self)
        elif method == self.METHOD_LOWERCASE:
            id = self.remote_py.lowercase(text, self)
        if id<0:
            self.status.setText(self.TEXT_ERROR)

    def onRemoteResponse(self, response, request_info):
        self.status.setText(response)

    def onRemoteError(self, code, message, request_info):
        self.status.setText("Server Error or Invalid Response: ERROR %d - %s" %
                            (code, message))


class EchoServicePython(JSONProxy):
    def __init__(self):
        JSONProxy.__init__(self, "/json", ["echo", "reverse", "uppercase", "lowercase"])

if __name__ == '__main__':
    # for pyjd, set up a web server and load the HTML from there:
    # this convinces the browser engine that the AJAX will be loaded
    # from the same URI base as the URL, it's all a bit messy...
    pyjd.setup("http://127.0.0.1/examples/jsonrpc/public/JSONRPCExample.html")
    app = JSONRPCExample()
    app.onModuleLoad()
    pyjd.run()

そしたらそのディレクトリに入ってコンパイルします。今回はターゲットを直
接 ${PROJECT_DIR}/media にしてしまいます。


$ cd kay-pyjamas-sample
$ ${PYJAMAS_HOME}/bin/pyjsbuild -o ${PROJECT_DIR}/media JSONRPCExample.py

成功すると ${PROJECT_DIR}/media/JSONRPCExample.html が出来ます。
http://localhost:8080/media/JSONRPCExample.html で画面が出ていればまあ成功です。

サーバーサイドは下記の通りにします。myapp はアプリケーション名に読み換
えてくださいね。

${PROJECT_DIR}/myapp/urls.py


# -*- coding: utf-8 -*-
# myapp.urls


from werkzeug.routing import (
  Map, Rule, Submount,
  EndpointPrefix, RuleTemplate,
)
import myapp.views

def make_rules():
  return [
    EndpointPrefix('myapp/', [
      Rule('/', endpoint='index'),
      Rule('/json', endpoint='json_rpc'),
    ]),
  ]

all_views = {
  'myapp/index': myapp.views.index,
  'myapp/json_rpc': myapp.views.json_rpc,
}

最後に view です。

${PROJECT_DIR}/myapp/views.py


# -*- coding: utf-8 -*-
# myapp.views
# ... 省略

import simplejson

# ... 省略

def json_uppercase(args):
  return [args[0].upper()]

def json_echo(args):
  return [args[0]]

def json_reverse(args):
  return [args[0][::-1]]

def json_lowercase(args):
  return [args[0].lower()]

def json_rpc(request):
  if request.method:
    args = simplejson.loads(request.data)
    method_name = 'json_%s' % args[u"method"]
    json_func = globals().get(method_name)
    json_params = args[u"params"]
    json_method_id = args[u"id"]
    result = json_func(json_params)
    args.pop(u"method")
    args["result"] = result[0]
    args["error"] = None
    return Response(simplejson.dumps(args), content_type="application/json")
  else:
    return Response('Error')

適当な作りですが、まあこれで動きます。興味があれば色々工夫してみてくだ
さい。

今日はここでおしまいです。

August 4, 2009

Kay framework を使ってみる

Kay framework を 7/7 にリリースしたわけですが、日本語での情報が無いので少しずつ書いていくことにしようと思います。

始め方

必要なものは下記の通りです。

Python-2.5
App Engine SDK Python
Kay framework

macports を使って python25 を入れた場合は、他に下記もインストールしましょう。

py25-hashlib
py25-socket-ssl
py25-pil
py25-ipython

Kay のリリース版を使う場合は、下記のダウンロードページから tar ball を落として使います。リポジトリを追いかける場合は、mercurial で clone してください。

ダウンロードページ: http://code.google.com/p/kay-framework/downloads/list

clone するには:


$ hg clone https://kay-framework.googlecode.com/hg/ kay

プロジェクトを始める

kay の manage.py スクリプトでプロジェクトディレクトリを作る事ができます。


$ python kay/manage.py startproject myproject
$ tree myproject
myproject
|-- app.yaml
|-- kay -> /Users/tmatsuo/work/tmp/kay/kay
|-- manage.py -> /Users/tmatsuo/work/tmp/kay/manage.py
|-- settings.py
`-- urls.py

1 directory, 4 files

シンボリックリンクをサポートしているプラットフォームでは kay ディレクトリと manage.py へのシンボリックリンクが作成されます。後で kay の場所を動かすときっと動かなくなるのですが、そんな時はリンクを張り直してください。

アプリケーションを作る

出来たばかりの myproject ディレクトリに cd して、早速アプリケーションを作りましょう。


$ cd myproject
$ python manage.py startapp myapp
$ tree myapp
myapp
|-- __init__.py
|-- models.py
|-- templates
|   `-- index.html
|-- urls.py
`-- views.py

1 directory, 5 files

こうして作成した myapp を動作させるにはもうひと手間必要です。settings.py の INSTALLED_APPS に登録します。必要なら APP_MOUNT_POINTS も登録します。下記の例では、アプリケーションをルート URL にマウントする例です。APP_MOUNT_POINTS を設定しない場合は、/myapp というようにアプリケーション名 URL にマウントされます。

settings.py


#$/usr/bin/python
#..
#..

INSTALLED_APPS = (
  'kay.sessions',
  'myapp',
)

APP_MOUNT_POINTS = {
  'myapp': '/',
}

見れば分かると思いますが、INSTALLED_APPS はタプルで、APP_MOUNT_POINTS は dict になっています。

アプリケーションを動かす

作ったアプリケーションを動かしてみましょう。下記のコマンドで開発サーバが起動する筈です。


$ python manage.py runserver
INFO     2009-08-04 05:48:21,339 appengine_rpc.py:157] Server: appengine.google.com
...
...
INFO     2009-08-04 05:48:21,448 dev_appserver_main.py:465] Running application myproject on port 8080: http://localhost:8080

この状態で http://localhost:8080/ にアクセスすると、「Hello」又は「こんにちは」と表示される筈です。

GAE にアップロードする

GAE にアップロードするには、対象の appid を app.yaml の application に設定してから、下記のコマンドを使用します。


$ python manage.py appcfg update

成功すると、http://your-appid.appspot.com/ でアクセスできるようになります。

今日はここまでにしますね。けっこう簡単に始められる事がお分かりいただけたでしょうか。次回はモデル定義とか、フォームの自動生成などについてお話しする予定です。

当分このシリーズを続けるつもりなので、取り上げて欲しいトピックなどがあればコメントください。

May 15, 2009

A patch for Werkzeug debugger on GAE

I wrote about using werkzeug on gae, but actually there are still several problems in debug console. So I wrote a patch for it.



diff -r 8ffe4e637e17 werkzeug/debug/console.py
--- a/werkzeug/debug/console.py Sat Apr 25 17:06:40 2009 +0000
+++ b/werkzeug/debug/console.py Fri May 15 00:40:32 2009 +0900
@@ -32,6 +32,12 @@
     def close(self):
         pass
 
+    def flush(self):
+        pass
+
+    def seek(self, n, mode=0):
+        pass
+
     def reset(self):
         val = ''.join(self._buffer)
         del self._buffer[:]
@@ -48,12 +54,18 @@
     def writelines(self, x):
         self._write(escape(''.join(x)))
 
+    def readline(self):
+        if len(self._buffer) == 0:
+            return ''
+        ret = self._buffer[0]
+        del self._buffer[0]
+        return ret
 
 class ThreadedStream(object):
     """Thread-local wrapper for sys.stdout for the interactive console."""
 
     def push():
-        if sys.stdout is sys.__stdout__:
+        if not isinstance(sys.stdout, ThreadedStream):
             sys.stdout = ThreadedStream()
         _local.stream = HTMLStringO()
     push = staticmethod(push)

May 14, 2009

Using Werkzeug's debugger on GAE dev server

There is a very good web framework called Werkzeug. The debugger of Werkzeug is pretty nice and we can use inline console on our web browser with it.

The author of Werkzeug wrote an article of how to use this debugger with GAE dev server. However according this article, the inline console is not usable.

After a little struggling, finally I can use the debugger on GAE dev server. GAE dev server creates an instance of DebuggedApplication on every request. I think its the reason why. So, it seems that specifying the instance as the module global singleton made it work.



_debugged_app = None
app = ... build your wsgi app ...

def main():
  # Only run the debugger in development.
  import os, sys
  if 'SERVER_SOFTWARE' in os.environ and os.environ['SERVER_SOFTWARE'].startswith('Dev'):
    # use our debug.utils with Jinja2 templates
    import debug.utils
    sys.modules['werkzeug.debug.utils'] = debug.utils

    # don't use inspect.getsourcefile because the imp module is empty 
    import inspect
    inspect.getsourcefile = inspect.getfile
    
    # wrap the application
    from werkzeug import DebuggedApplication
    global _debugged_app
    if _debugged_app is None:
      _debugged_app = app = DebuggedApplication(app, evalex=True)
    else:
      app = _debugged_app

Google App Engine で Werkzeug のデバッガを使う

Werkzeug という Python Web Framework があります。Werkzeug のデバッガはなかなかすぐれもので Web アプリでエラーが出るとインライン console が使えたりします。

このデバッガを GAE の dev server で使う方法を作者の方が書いているのですが、この通りやるとインライン console が使えなかったりします。

ちょっと調べてみたら意外と簡単に使えました。GAE の dev server だとリクエスト毎に DebuggedApplication のインスタンスが作り直されてしまうので、これを module global のシングルトンにしてやればオッケイでした。



_debugged_app = None
app = ... build your wsgi app ...

def main():
  # Only run the debugger in development.
  import os, sys
  if 'SERVER_SOFTWARE' in os.environ and os.environ['SERVER_SOFTWARE'].startswith('Dev'):
    # use our debug.utils with Jinja2 templates
    import debug.utils
    sys.modules['werkzeug.debug.utils'] = debug.utils

    # don't use inspect.getsourcefile because the imp module is empty 
    import inspect
    inspect.getsourcefile = inspect.getfile
    
    # wrap the application
    from werkzeug import DebuggedApplication
    global _debugged_app
    if _debugged_app is None:
      _debugged_app = app = DebuggedApplication(app, evalex=True)
    else:
      app = _debugged_app

October 11, 2008

A custom widgets for GeoPt Property

I attended a Maps API hack-a-thon in Tokyo. There I wrote some utility class which helps us to handle GeoPt Property very easy.

In this article, I'll show you some sample code which store geopoint data by just clicking on Google Map. Before we go, perhaps you can read the djangoforms documentation. This will help you to understand the codes in this article.

Please create a directory for this article, copy my_geopt.py to the directory and create 3 brand new files app.yaml, main.py, and index.html. That's all.

After creating these files, start local dev server, get access to localhost, and see that you can specify a particular GeoPt by just clicking on the Google Map.

And here is a live demo(slightly more complicated, but basically almost the same).

Happy coding :-)

app.yaml

application: geotest
version: 1
runtime: python
api_version: 1

handlers:
- url: /.*
  script: main.py

main.py

from google.appengine.ext import webapp
from google.appengine.ext.webapp.util import run_wsgi_app

import os
import wsgiref.handlers

from google.appengine.ext import db
from google.appengine.ext.webapp import template
from google.appengine.ext.db import djangoforms
import my_geopt

class Place(db.Model):
  name = db.StringProperty()
  description = db.StringProperty()
  geopoint = db.GeoPtProperty()
  date = db.DateTimeProperty(auto_now=True)

class PlaceForm(djangoforms.ModelForm):
  class Meta:
    model = Place

class MainPage(webapp.RequestHandler):
  def post(self):
    data = PlaceForm(data=self.request.POST)
    if data.is_valid():
      # Save the data, and redirect to the view page
      entity = data.save(commit=False)
      entity.put()
      self.redirect('/')
    else:
      # Reprint the form
      contents = {'form': data}
      path = os.path.join(os.path.dirname(__file__), 'index.html')
      self.response.out.write(template.render(path, contents))
  def get(self):
    contents = {'form': PlaceForm()}
    path = os.path.join(os.path.dirname(__file__), 'index.html')
    self.response.out.write(template.render(path, contents))

application = webapp.WSGIApplication(
  [('/', MainPage),],
  debug=True)

def main():
  run_wsgi_app(application)

if __name__ == "__main__":
  main()

index.html (Please replace 2 YOURAPIKEYs with your Maps API Key)

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<title>GeoPtWidgets Example</title>
<script src="http://maps.google.com/maps?file=api&v=2&key=YOURAPIKEY"
type="text/javascript"></script>
<script src="http://www.google.com/uds/api?file=uds.js&v=1.0&key=YOURAPIKEY"
type="text/javascript"></script>
<script src="http://www.google.com/uds/solutions/localsearch/gmlocalsearch.js"
type="text/javascript"></script>
<style type="text/css">
@import url("http://www.google.com/uds/css/gsearch.css");
@import url("http://www.google.com/uds/solutions/localsearch/gmlocalsearch.css");
</style>
</head>

<body onunload="GUnload()">
<h1>GeoPtWidgets Example</h1>
<form method="post" action="/">
{{ form.as_p }}
<input type="submit" value="register"/>
</form>
</body> </html>

July 13, 2008

Using the newest zipped pytz on GAE

I posted an entry about how to use zipped pytz on GAE[1].

1. http://takashi-matsuo.blogspot.com/2008/07/using-zipped-pytz-on-gae.html

It works well with older versions of pytz. Stefano pointed out that my method doesn't work with the newest pytz distribution. This article describes how to use the newest zipped pytz on GAE.

First, retrieve the newest pytz from pypi[2] and extract the archive.

2. http://pypi.python.org/pypi/pytz/


$ tar xjf pytz-2008c.tar.bz2
$ cd pytz-2008c/pytz
$ zip -q zoneinfo.zip `find zoneinfo -type f ! -name '*.pyc' -print`
$ rm -rf zoneinfo

After that, you have to edit pytz/__init__.py and modify open_resource function to use zipped zoneinfo database like following:
(pkg_resources stuff were struck out. Thank you again Stefano!)


def open_resource(name):
"""Open a resource from the zoneinfo subdir for reading.

Uses the pkg_resources module if available.
"""
    import zipfile
    from cStringIO import StringIO

if resource_stream is not None:
    return resource_stream(__name__, 'zoneinfo/' + name)
else:
    name_parts = name.lstrip('/').split('/')
    for part in name_parts:
        if part == os.path.pardir or os.path.sep in part:
            raise ValueError('Bad path segment: %r' % part)
    zoneinfo = zipfile.ZipFile(os.path.join(os.path.dirname(__file__),
        'zoneinfo.zip'))
    return StringIO(zoneinfo.read(os.path.join('zoneinfo', *name_parts)))

Then, the only thing to do is to copying your original pytz directory into your application directory.


$ cd ../
$ cp -r pytz /your/application/directory

If you'd like to avoid using CPU to unzip zoneinfo data every time, perhaps you could use following function:


from google.appengine.api import memcache
import logging
import pytz
from pytz import timezone

def getTimezone(tzname):
  try:
    tz = memcache.get("tz:%s" % tzname)
  except:
    tz = None
    logging.debug("timezone get failed: %s" % tzname)
  if tz is None:
    tz = timezone(tzname)
    memcache.add("tz:%s" % tzname, tz, 86400)
    logging.debug("timezone memcache added: %s" % tzname)
  else:
    logging.debug("timezone memcache hit: %s" % tzname)

  return tz

Happy coding :-)

July 12, 2008

Using zipped pytz on GAE

To handle timezone of all over the world crrectly, I think it is the most easy way to use pytz[1]. Besides that, there are many modules and programs which depends on pytz modes, so I believe that quite a few people would like to use pytz on GAE. However, pytz contains over 500 files in its zoneinfo directory, so it often bears on the 1000 files limit.

1. http://pytz.sourceforge.net/

This is an instruction about how to use zipped pytz on GAE environment. In this article, we use my_zipimport.py[2] provided by Guido van Rossum.

2. http://code.google.com/p/googleappengine/issues/detail?id=161#c19

First, make your own pytz.zip as following:


$ cd pytz-2006p
$ zip -q pytz.zip `find pytz -type f ! -name '*.pyc' -print`

Secondly, make sure to put my_zipimport.py and pytz.zip into your GAE app directory.
Lastly, put following code into your script.


import my_zipimport
import sys

my_zipimport.install()
sys.path.insert(0, 'pytz.zip')

That's all. Now you can use pytz module seamlessly.

June 27, 2008

グループオーナー (続き)

python-ml-jp にて Google-App-Engine-Japan グループを宣伝したら 30 人ほどメンバーが増えました!
みなさんようこそいらっしゃいました。

メンバーも増えた亊だし少し語りかけてみようかなと思います。

June 1, 2008

My first app on App Engine

I attended the conference named 'Google I/O'. It was a very exciting event for an engineer like me. I attended many sessions that were, in most cases, related to Google App Engine.

Google App Engine is a web platform where we can run our web application programs on Google's infrastructure. In my opinion, that could be the biggest paradigm shift this decade.

In those sessions, I learned many techniques about how to build applications on Google App Engine, so I finally wrote up my first application on App Engine called 'InterNovel'. The URL is below.

http://internovel.appspot.com/

In this application, there are novels. Each novel has its own state. The state is one of the following three: 1. Accepting fragments. 2. Votes. 3. Complete.

1. Accepting fragments
When the novel is in this state, everyone can post a continuous fragment. If the number of posted fragments reaches a particular count (that the owner of the novel decided beforehand), the state will change to 2.

2. Votes
When the novel is in Votes state, everyone can vote on their favorite fragments among the candidates. If the total number of the votes reaches a particular count (this number is also editable by the owner), the state will change to 1 or 3.

3. Complete
If the number of iterations between states 1 and 2 reaches a particular number, the completion comes. The state will never change from this time.

Well, that's all about this application. Isn't it interesting?

May 30, 2008

Google I/O が終わりました

終わってしまいました。 Guido のセッションがものすごく High Level だったのが意外でした。 High Level というのは「 core では無い」という意味ですが。

全体的な感想ですが、今回は主な興味の対象である App Engine のセッションを中心に見てきまして、 App Engine についていろいろあやふやだった点がはっきりしてきました。とても有意義なセッション達でした。

ところで今回同行した a2c さんが面白い亊を言っていました。興味の無いテクノロジーのセッションを聞くのも意外と楽しい...と。そういうものかもしれません。発見する亊はそもそも楽しいですからね。

そうそう。以前のポストで書いた App Engine 用のソフトウェアがだいたい完成しましたので、URL を書いときます。
http://internovel.appspot.com/

まだ説明とか全然足りないので、多分正体や使い方も皆目分からないと思います。それらは日本に帰ってからおいおい書くつもりです。

今日はとりあえずこんなところで。

May 25, 2008

Google I/O前夜

Google I/O が近づいて来ましたので準備をはじめました。まずはプログラムのスケジュールをダウンロードして印刷しておきます。これを飛行機の中で眺めておけば、参加したいプログラムをある程度しぼれるでしょう。荷物は少なめです。いつも使っているトラベルバッグに収まりそう。

Google I/O 自体は 5/28, 5/29 の2日間なのですが、私自身は 5/27 のうちに San Francisco に到着予定です。この日の夕食をどうするか悩んでいたのですが、何人かの Googler がホストしてくれる亊になったので気が楽になりました :-)

私個人として、今一番注目しているテクノロジーは App Engine です。Google のインフラを使って Web アプリが動かせるのですから、冗長化の面でもスケーラビリティの面でも大きなメリットがあります。それに python だし。 python は私のお気に入りです。

そうそう。まだ内容は内緒ですが、 App Engine で動く Web アプリを作っている最中なのです。もう少しでプロトタイプが完成しそうなので、飛行機の中で書き終えられたら良いなと思っています。とりあえず動くものが完成したら App Engine Group にポストしてみるつもりです。(5/26追記) グループにポストするよりも Applications Gallery に登録する方が良さそうですね。

June 7, 2007

First post

I had finished to write Google Apps Provisioning API V2 client codes in python. I wish they will take it in the main tree. I have some confidence about that.

I am writing tgsso too. What is tgsso? Tgsso is TurboGears Google Single Sign On. It's a little web application which enables you to authenticate Google Apps users by PAM. Its outline is almost done, but there are many things to do.

By the way, tomorrow I and my wife will have complete medical checkup. I hope we both are in good health. So, we are banned from eating or drinking anything after 9pm. Sigh...

Let's sleep early.

November 2, 2009

October 21, 2009

Overview

How to test

Results

Cold Starting result

webapp+django template

Kay

app-engine-patch

Served by hot instances result

webapp+django template

Kay

app-engine-patch

The application used in these tests

webapp+django template

Kay

app-engine-patch

Conclusion

Cold Starting result

tipfy

Served by hot instances result

tipfy

Application

tipfy

August 31, 2009

JSON-RPC Example

JSON-RPC Example

August 4, 2009

始め方

プロジェクトを始める

アプリケーションを作る

アプリケーションを動かす

GAE にアップロードする

May 15, 2009

May 14, 2009

October 11, 2008

July 13, 2008

July 12, 2008

June 27, 2008

June 1, 2008

May 30, 2008

May 25, 2008

June 7, 2007

About Me

Blog Archive