May 14, 2010

Now Kay Framework has GAETestBase bundled!

What I've been working on recently is to incorporate GAETestBase to Kay Framework.

I've just finished the adapting work and bundled GAETestBase works very nicely with kay framework. You just need to change the base class of your testcase to "kay.ext.testutils.gae_test_base.GAETestBase" like:

from kay.ext.testutils.gae_test_base import GAETestBase
class DummyTest1(GAETestBase):
  #...
  def test_foo(self):
    # ...
    # ...

By default, when you run your tests via gaeunit, the web testrunner will collect testcases from all of your applications defined in settings.INSTALLED_APPS variable. For now, you cannot change this behavior, but in the near future, I'll add a feature for configuring which test to run via the gaeunit web testrunner.

The recent version of Kay has a url mapping for web testrunner, but your app.yaml may lack of it. In such a case, please add following entry to your app.yaml:

- url: /_ah/test.*
  script: kay/ext/testutils/gaeunit.py
  login: admin

As you can see, you can run your test by visiting a URL like "/_ah/test".

As a note, all of your tests will be called in parallel, so it might leads to unexpected test failures. So you need to write your test functions independent from other tests.

Please see my previous post for details about how to configure GAETestBase.

Happy testing!

How to download souce code of your application written with kay

Let me introduce a small trivia to you. What if your harddisk crashes and you lost your source code, hopefully, you might use some source code management system such as cvs, subversion, mercurial or git, in such a case, you can retrieve your source code back. But what if the crash occurs before your first commit, or your repository also crashes?

Don't give up!

Recent version of Kay Framework has remote_api and deferred handler by default, so there is still a chance  for retrieving it back to you.

This idea is originally come from this thread in stackoverflow.

  • First, create a brand new project with kay's manage.py script and edit your app.yaml. Just put your application-id on the top of the file.

    $ python ~/work/kay/manage.py startproject downloadcode
    Running on Kay-0.10.0
    Finished creating new project: downloadcode.
    $ cd downloadcode
    $ vi settings.py
    
  • Second, create a file named "restore_code.py" as follows:

    import os
    from google.appengine.ext import db
    
    expr = """
    [type(
        'CodeFile',
        (__import__('google.appengine.ext.db').appengine.ext.db.Expando,),
        {})(
            name=dp+'/'+fn,
            data=__import__('google.appengine.ext.db').appengine.ext.db.Text(
                open(dp + '/' + fn).read()
            )
        ).put() if not dp.startswith('../kay') else None
     for dp, dns, fns in __import__('os').walk('..')
     for fn in fns]
    """
    
    
    class CodeFile(db.Model):
      name = db.StringProperty(required=True)
      data = db.TextProperty(required=True)
    
      @classmethod
      def kind(cls):
        return cls.__name__
    
    def restore():
      for cf in CodeFile.all():
        fpath = cf.name.replace("../", "./_restored_files/")
        dirname = os.path.dirname(fpath)
        if not os.path.exists(dirname):
          os.makedirs(dirname)
        fh = open(fpath, "w")
        fh.write(cf.data)
        fh.close()
    
    def scan():
      from google.appengine.ext.deferred import defer
      defer(eval, expr)
    
  • Next, connect production server with "python manage.py rshell command."
    $ python manage.py rshell
    Running on Kay-0.10.0
    Username:matsuo.takashi
    Password:
    Interactive Kay Shell with RemoteDatastore. 
    -----------------WARNING--------------------
    
    Please be careful in this console session.
    
    -----------------WARNING--------------------
    
    
    In [1]: import restore_code
    In [2]: restore_code.scan()
    restore_code.scan() will scan filesystems on the production and create CodeFile entities in the datastore. This work will be done by deferred handler, so you may need to wait a bit, but the completion of this process won't be notified. So why don't you take a break here and have a cup of coffee.
  • Lastly, call "restore_code.restore()" function.
    In [3]: restore_code.restore()
    In [4]:

Then, you can have your entire source code except for static files and configuration files(such as app.yaml, index.yaml, etc..) in a directory named "_restored_files".

Yey, now you've got your code back!

May 13, 2010

Introducing GAETestBase

A Japanese highly skilled engineer tago-san wrote an excellent module named GAETestBase. I'd like to introduce GAETestBase to you all. This entry is basically an English translation of this article of his.

GAETestBase is a useful subclass of the standard unittest.TestCase. You can just write your test classes as usual by just extending GAETestBase class for utilizing this modules functionality.

GAETestBase can be downloaded at Google Code Repository.

What can I do with this module?

  • Run your tests via CLI
    • You can run your tests without setting up necessary environments (e.g. stubs,  env, etc..)
    • Tests will be use remote_api connection for accessing production datastore if you configure to do so in your TestCase.
  • Run your tests via GAEUnit
    • GAEUnit is a web-based test runner hosted at: GAEUnit Google Code
    • You can run your test with web-browser in your development environment.
    • You can also run your tests with web-browser in your production environment. The test will be invoked under real production services(not memory based DatastoreFileStub) if you configure to do so.
  • Run your tests with special overridden kind() method.
    • You can use this overridden kind method just in your tests without any changes in your code.
      • For example, entities of  a model class "MyModel" will be stored as kind "t_MyModel".
      • Be careful with this special kind method in your TestCase. Please see an example bellow.

        Key.from_path('MyModel', id)       # NG!
        Key.from_path(MyModel.kind(), id)  # OK!
    • Especially useful with tests via remote_api and tests on production(GAEUnit) because you can run your tests without any data pollution.
    • Of course, you can suppress this behavior by configuration
  • Clean up all the kinds that are used in your tests.
    • You can delete all the kinds used in your test after running your test.
      • Only the kinds which is accessed in a particular TestCase, will be deleted.
      • *accessed* here includes just reading. So if you read a existing kind, the kind will become a target of deletion.
    • CAUTION: This feature could be very dangerous if you disable overridden kind method.

    Let's do it

    Download necessary files and deployment

    You need to do following three things before start writing your tests.
    1. Deploy gaeunit.py
      • Download the file and put it into your project directory.
      • You need to re-write this file for changing _LOCAL_TEST_DIR if your tests are not placed in test/*.py.
    2. Add two entries to your app.yaml
      • Configure a gaeunit handler and remote_api handler as follows:


        - url: /remote_api
          script: $PYTHON_LIB/google/appengine/ext/remote_api/handler.py
          login: admin
        - url: /test.*
          login: admin  # This is important if you deploy the test directory in production!
          script: gaeunit.py

    3. Create "test" directory and put gae_test_base.py into it, and edit constant variables in the file.
      • You need to configure following variables:
        • GAE_HOME: a path for google appengine SDK
        • PROJECT_HOME: a path for your target project
        • APP_ID: your application id
        • REMOTE_API_ENTRY_POINT: a path section of the URI of remote_api (e.g. "/remote_api" for above example).

    Run your test in local env.

    Firstly, let's try running your test for local environment. This is a pretty ordinary test. "test_defs" here is just for model definition, so replace it with actual module with yours.

    from gae_test_base import *
    
    from test_defs import *
    from google.appengine.ext import db
    
    class DummyTest1(GAETestBase):
    
        def test_put(self):
            x1 = XEntity(x="x1")
            k1 = x1.put()
            self.assertEqual(db.get(k1).x, "x1")
    
        def test_tx(self):
            def tx1(x2key, x2, x3):
                x2 = XEntity(key=x2key, x=x2)
                x3 = XEntity(x=x3, parent=x2)
                x2.put()
                x3.put()
                return (x2.key(), x3.key())
            x2k = db.allocate_ids(db.Key.from_path(XEntity.kind(), -1), 1)[0]
            k2, k3 = db.run_in_transaction(tx1, db.Key.from_path(XEntity.kind(), int(x2k)), "x2", "x3")
            self.assertEqual(db.get(k2).x, "x2")
            self.assertEqual(db.get(k3).x, "x3")

    Actual tests are very simple as follows:
    1. test_put
      • Create a new entity and put() it.
      • Getting an entity with returned key again, and compare values of two entities.
    2. test_tx
      • Define a following function "tx1" for transaction
        • receives a key for a root entity, a value for a property of the root entity, and a value for a property of the child entity as its arguments.
        • creates a new root entity and a child entity.
        • puts'em together
        • returns those two keys
      • Get an id for a new root entity by invoking allocate_ids
      • Run tx1 under a transaction
      • Get two entities with the key returned from transaction call, and compare property values with expected one.
    Here is an example output when running this testcase:

    $ python2.5 dummy_test.py 
    ..
    ----------------------------------------------------------------------
    Ran 2 tests in 0.020s
    
    OK
    $

    Here is an example screenshot when running this testcase via GAEUnit:


    Run your test via remote_api or on production

    Next, let's add a new testcase that uses the production datastore as its backend. Here is an example configuration for using remote_api via CLI, and using production datastore when deployed on appspot.

    class DummyTest2(GAETestBase):
        USE_PRODUCTION_STUBS = True
        USE_REMOTE_STUBS = True
    
        def test_put(self):
            y1 = YEntity(y="y1")
            k1 = y1.put()
            self.assertEqual(db.get(k1).y, "y1")
    
        def test_tx(self):
            def tx1(y2key, y2, y3):
                y2 = YEntity(key=y2key, y=y2)
                y3 = YEntity(y=y3, parent=y2)
                y2.put()
                y3.put()
                return (y2.key(), y3.key())
            y2k = db.allocate_ids(db.Key.from_path(YEntity.kind(), -1), 1)[0]
            k2, k3 = db.run_in_transaction(tx1, db.Key.from_path(YEntity.kind(), int(y2k)), "y2", "y3")
            self.assertEqual(db.get(k2).y, "y2")
            self.assertEqual(db.get(k3).y, "y3")

    You can configure all the behavior by defining class attributes on your extended TestCase classes. The default value of all the configurable attributes is False when omitted. This new testcase is almost the same as the previous testcase except for a target model class.

    Here is an example output via CLI:

    $ python2.5 dummy_test.py 
    ....
    ----------------------------------------------------------------------
    Ran 4 tests in 6.578s
    
    OK
    $

    Now you can recognize that, this time, two testcases are called. You may asked your username and password on test that uses remote_api, though its omitted because there is a valid appcfg_cookies.

    Here is an output via GAEUnit:


    You can also run these tests on appspot by just visiting http://app-id.appspot.com/test. After running these tests on appspot, you can see that entities which are used in your test are stored in the production datastore. A kind of these entities is "t_YEntity" (specially prefixed for test) like below:


    Cleaning up entities in your tests

    If you don't like entities created in your tests remains, you can write a testcase which deletes entities which are accessed in your tests like:

    class DummyTest3(GAETestBase):
        USE_PRODUCTION_STUBS = True
        USE_REMOTE_STUBS = True
        CLEANUP_USED_KIND = True
    
        def test_put(self):
            z1 = ZEntity(z="z1")
            k1 = z1.put()
            self.assertEqual(db.get(k1).z, "z1")
    
        def test_tx(self):
            def tx1(z2key, z2, z3):
                z2 = ZEntity(key=z2key, z=z2)
                z3 = ZEntity(z=z3, parent=z2)
                z2.put()
                z3.put()
                return (z2.key(), z3.key())
            z2k = db.allocate_ids(db.Key.from_path(ZEntity.kind(), -1), 1)[0]
            k2, k3 = db.run_in_transaction(tx1, db.Key.from_path(ZEntity.kind(), int(z2k)), "z2", "z3")
            self.assertEqual(db.get(k2).z, "z2")
            self.assertEqual(db.get(k3).z, "z3")

    You can run these test via CLI, via GAEUnit(dev), and via GAEUnit(prod).

    After running your tests, you can see there is no "t_ZEntity" on the datastore at all:


    Per testcase cutomizing

    Here is an example for customizing kind prefix and clean up behavior.


    class DummyTest4(GAETestBase):
        USE_PRODUCTION_STUBS = True
        USE_REMOTE_STUBS = True
        CLEANUP_USED_KIND = False
        KIND_PREFIX_IN_TEST = 'test'
    
        def test_put(self):
            z1 = ZEntity(z="z1")
            k1 = z1.put()
            self.assertEqual(db.get(k1).z, "z1")
    
        def test_tx(self):
            def tx1(z2key, z2, z3):
                z2 = ZEntity(key=z2key, z=z2)
                z3 = ZEntity(z=z3, parent=z2)
                z2.put()
                z3.put()
                return (z2.key(), z3.key())
            z2k = db.allocate_ids(db.Key.from_path(ZEntity.kind(), -1), 1)[0]
            k2, k3 = db.run_in_transaction(tx1, db.Key.from_path(ZEntity.kind(), int(z2k)), "z2", "z3")
            self.assertEqual(db.get(k2).z, "z2")
            self.assertEqual(db.get(k3).z, "z3")

    After running this test, you can see "test_ZEntity" is stored in the datastore(ruled out from deletion).


    Accessing the real data on production


    Here is an example testcase for accessing the real data on prod.


    class DummyTest5(GAETestBase):
        USE_PRODUCTION_STUBS = True
        USE_REMOTE_STUBS = True
        CLEANUP_USED_KIND = False
        KIND_NAME_UNSWAPPED = True
    
        def test_put(self):
            count = PEntity.all().count()
            p1 = PEntity(i=count)
            k1 = p1.put()
            px = db.get(k1)
            self.assertEqual(px.i, count)

    If KIND_NAME_UNSWAPPED is set to True, you can suppress overriding kind method on db.Model. So, if you turn on USE_REMOTE_STUBS or USE_PRODUCTION_STUBS, your tests can access the real data on the production datastore.

    How to use this module with Kay Framework?

    Now Kay has GAETestBase bundled, so you can use it out of the box!
    For more details, please see this article as well.

    Caution

    TODO: translate


    Far beyond

    TODO: translate

    May 8, 2010

    Creating an app for Google Apps Marketplace with Kay - part 1 (revised)

    This entiry is obsoleted because Appengine SDK-1.3.4 was released, and it has a capability for OpenID authentication. I'll definitely start figuring an easy way for writing Google App Engine application for Google Apps Marketplace soon, so please wait a bit.

    I've just implemented an authentication mechanism in Kay framework for Google Apps MarketPlace. So let me introduce how to deploy a marketplace application with Kay(repository version).

    I have made a few more improvements in authentication system of Kay Framework, so I slightly change this article at 4:20PM on May 13th 2010JST.

    1. Create an app-id

    In this example, I've created a brand new application slot called 'marketplace-app'.

    2. Register a listing on the MarketPlace

    Firstly, you need to register yourself as a MarketPlace Vendor.
    http://www.google.com/enterprise/marketplace/

    Click a link 'Become a vendor' in the bottom of the page above and you'll get an instruction.

    Actually you need to click a link 'Sign in' at top-right corner of the page above, and fill the form.

    Then, you can create your listing. Click a button 'Create a new listing' in your Vendor Profile page.

    In this example, lets create an installable app, so check the first checkbox 'My product may be directly installed into Google Apps domains'.

    You can fill other fields arbitrarily except for the field 'Manifest'.

    Manifest xml here:

    <?xml version="1.0" encoding="UTF-8" ?>
    <ApplicationManifest xmlns="http://schemas.google.com/ApplicationManifest/2009">
        
      <!-- Support info to show in the marketplace & control panel -->
      <Support>
        <!-- URL for application setup as an optional redirect during the install -->
        <Link rel="setup" href="https://marketplace-app.appspot.com/a/${DOMAIN_NAME}/setup" />
        
        <!-- URL for application configuration, accessed from the app settings page in the control panel -->
        <Link rel="manage" href="https://marketplace-app.appspot.com/a/${DOMAIN_NAME}/admin" />
    
        <!-- URL explaining how customers get support. -->
        <Link rel="support" href="https://marketplace-app.appspot.com/a/${DOMAIN_NAME}/support" />
      
        <!-- URL that is displayed to admins during the deletion process, to specify policies such as data retention, how to claim accounts, etc. -->
        <Link rel="deletion-policy" href="https://marketplace-app.appspot.com/deletion-policy" />
      </Support>
        
      <!-- Name and description pulled from message bundles -->
      <Name>Marketplace sample application</Name>
      <Description>A simple application for the marketplace</Description>
      
      <!-- Show this link in Google's universal navigation for all users -->
      <Extension id="navLink" type="link">
        <Name>Sample</Name>
        <Url>https://marketplace-app.appspot.com/a/${DOMAIN_NAME}/start</Url>
        <!-- This app also uses the Calendar API -->
        <Scope ref="calendarFeed"/>
      </Extension>
      
      <!-- Declare our OpenID realm so our app is white listed -->
      <Extension id="realm" type="openIdRealm">
        <Url>https://marketplace-app.appspot.com/</Url>
      </Extension>
    
      <!-- Need access to the Calendar feed -->
      <Scope id="calendarFeed">
        <Url>https://www.google.com/calendar/feeds/</Url>
        <Reason>This application shows the next Calendar event.</Reason>
      </Scope>
    
    </ApplicationManifest>

    In this example, there is a calendar feed setting for 2 legged OAuth access scope. For more details with a Manifest file format, please see:
    http://code.google.com/googleapps/marketplace/manifest.html

    After adding your listings, you will see a preview page for your app. There is an attractive 'Add it now' button on the right, but of course, you need to implement your application before adding this app.

    Instead, click a link 'My Vendor Profile' on the top-right corner and you can see your listings with a link 'View OAuth Consumer Key'.

    3. Implementation

    Let's create a new project with Kay management script:

    $ python /some/where/kay/manage.py startproject marketplace-app
    Running on Kay-0.10.0
    Finished creating new project: marketplace-app.
    $ cd marketplace-app
    $ python manage.py startapp core

    settings.py:

    INSTALLED_APPS = (
      'core',
      'kay.ext.gaema',
    )
    
    APP_MOUNT_POINTS = {
      'core': '/',
    }
    
    MIDDLEWARE_CLASSES = (
      'kay.sessions.middleware.SessionMiddleware',
      'kay.auth.middleware.AuthenticationMiddleware',
    )
    AUTH_USER_BACKEND = "kay.auth.backends.gaema.GAEMABackend"
    GAEMA_USER_MODEL = "core.models.User"
    
    GAEMA_SECRETS = {
      'google_consumer_key': 'your consumer key here',
      'google_consumer_secret': 'your consumer key secret here',
    }
    IS_MARKETPLACE_APP = True
    

    Replace GAEMA_SECRETS with actual values. You can see consumer key and consumer key secret with clicking the link 'View OAuth Consumer Key' mentioned above. The last attribute 'IS_MARKETPLACE_APP' is important for this authentication system work normaly.

    core/models.py:

    # -*- coding: utf-8 -*-                                                                                                                                          
    # core.models                                                                                                                                                    
    
    from google.appengine.ext import db
    from kay.ext.gaema.models import GAEMAUser
    
    # Create your models here.                                                                                                                                       
    
    class User(GAEMAUser):
      pass
    

    Information of users will be stored into this model.

    core/urls.py:

    # -*- coding: utf-8 -*-
    # core.urls
    #
    
    from kay.routing import (
      ViewGroup, Rule
    )
    
    view_groups = [
      ViewGroup(
        Rule('/a/<domain_name>/setup', endpoint='domain_setup',
             view='core.views.domain_setup'),
    
        Rule('/a/<domain_name>/admin', endpoint='domain_admin',
             view='core.views.domain_admin'),
    
        Rule('/a/<domain_name>/support', endpoint='domain_support',
             view='core.views.domain_support'),
    
        Rule('/a/<domain_name>/start', endpoint='domain_start',
             view='core.views.domain_start'),
    
        Rule('/deletion_policy', endpoint='deletion_policy',
             view='core.views.deletion_policy'),
      )
    ]
    

    core/views.py:

    # -*- coding: utf-8 -*-                                                                                                                                          
    """                                                                                                                                                              
    core.views                                                                                                                                                       
    """
    from werkzeug import (
      unescape, redirect, Response,
    )
    
    from kay.utils import render_to_response
    from kay.ext.gaema.utils import get_gaema_user
    from kay.auth.decorators import login_required
    
    def index(request):
      return render_to_response('core/index.html', {'message': 'Hello'})
    
    def domain_setup(request, domain_name):
      callback = request.args.get('callback')
      if callback is None:
        return Response("No callback supplied.")
      return redirect(callback)
    
    def domain_admin(request, domain_name):
      return Response("%s administration." % domain_name)
    
    def domain_support(request, domain_name):
      return Response("%s support." % domain_name)
    
    def deletion_policy(request):
      return Response("Deletion policy.")
    
    @login_required
    def domain_start(request, domain_name):
      return Response("%s start.\n%s" % (domain_name, request.user.raw_user_data))
    

    A decorator 'login_required' takes care of OpenID/OAuth stuff, so you don't need to do anything at all. The user information is in request.user.raw_user_data.

    Let's deploy it to appspot.

    $ python manage.py appcfg update

    You can check the application with visiting https://marketplace-app.appspot.com/a/example.com/start where example.com should be replaced with your actual domain. This domain must be registered with Google Apps and make sure that "Federated Login using OpenID" is turned on.

    If the app is successfully deployed, you can see an OpenID authentication display. Let's move on.

    4. Add your app to your domain

    Go back to your Vendor Profile Page, and click your application title, then you will see the preview page of your application.

    Let's click 'Add it now' button on the right, and enter your google apps domain, and click 'Go'.
    Follow an adding application wizard like 1) Agree to terms 2) Grant data access 3) External configuration 4) Enable the app, and wait for a while, and you can see a universal navigation link 'Sample' on the top left corner of your google apps applications.

    If you click the link, you can silently signed into your application because Marketplace apps are whitelisted with a particular openid_realm.

    In part 2 of this article, I'll show you how to access user's calendar data with gdata library.

    To be continued..