Google App Engine on Docker for Development

If you like to use docker for development and you want to run google app engine as a container here is a short tutorial.

This example uses google app engine for python.

Docker Image

First of all we need a Dockerfile to describe the image:


# start from a python 2 official image
FROM python:2
# install unzip, download and unzip google app engine
RUN apt-get update \
&& apt-get install -y unzip \
&& wget https://storage.googleapis.com/appengine-sdks/featured/google_appengine_1.9.40.zip -P /tmp \
&& unzip -q /tmp/google_appengine_1.9.40.zip -d /opt > /dev/null
# expose application and admin port
EXPOSE 8080 8000
# start google app engine when the container is ran
CMD /opt/google_appengine/dev_appserver.py –host=0.0.0.0 –skip_sdk_update_check=yes –storage_path /usr/share/app/.data /usr/share/app

The Dockerfile should be saved on the root of your project, but don’t name it just Dockerfile otherwise app engine will try to run a container. We will name it project_dockerfile.

Now, from the root of our project, we can build an image out of the Dockerfile:

docker build -t project_image -f project_dockerfile .

Run the container

To run a container based on the freshly built image:

docker run -d -v ${PWD}:/usr/share/app -p 8080:8080 -p 8000:8000 --name=project
project_image
-d

tells docker to run in daemon mode

-v ${PWD}:/usr/share/app

mounts the current folder to the container’s /usr/share/app folder

-p 8080:8080 -p 8000:8000

forwards the 8080 and 8000 ports to the host 8080 and 8000 ports

Install python dependencies

If you need to install python dependencies and your project has a requirements.txt file,

connect to the running container

docker exec -it project /bin/bash

cd to the application folder

cd /usr/share/app

install the dependencies

pip install -r requirements.txt -t lib

and exit

exit

Monitor the application

Of course we want to read the applications log as we test it, we can do this by “attaching” to the running container:

docker attach project

as soon as the application logs we will see it on the console

But.. where is the application?

If you point your browser to 127.0.0.1:8080 you should find your application running.

127.0.0.1:8000 is the address to the google app engine’s administrative console

Data

Google app engine saves the application data to /usr/share/app/.data. Since this is mounted from the project root folder on our host, we will find the .data folder there as well.

This allows the google app engine to find the application data again after the container is stopped and restarted.

Stockholm Bike Tour: Täby – Vaxholm – Gustavsberg

Distance

100km ca.

Surface

asphalt

Course

Description

This is a quite famous bike tour for bikers starting from Stockholm.

Getting to Täby can be quite tricky if you don’t know the place and you want to ride on bicycle paths. Anyway there are signs guiding you to Täby.

Passed Rybdo you need to cycle for some 4kms on a big road where cars drive pretty fast. There is no other way to get closer to Vaxholm from there but you can use the emergency lane when it is available and in Sweden car drivers are very kind to bikers.

Once out of the big road you can ride quietly for 10kms on a typical swedish road sorrounded by woods and with few cars passing. This road will take you to Vaxholm. There you can take a ferry which will bring you to the island of Rindö. The crossing takes about 5 minutes (time tables and other info here).

On the other side of Rindö another ferry will take you back on the “mainland”. From there you can enjoin other 15kms of quiet road which will take you close to Gustavsberg.

From Gustavsberg the way back to Stockholm is mainly on bicycle paths.

Automatically Backup MySQL databases to Amazon S3

If you happen to run your own server you need to think about backups.

mysqlFor MySQL to backup a database is as simple as:

mysqldump -u user -ppassword my_database > /var/local/backups/mysql/my_database_dump.sql

Now you have a .sql file, which contains your database, stored into the /var/local/backups/mysql/ folder.

Is your database safe now? Well, no! What if your server dies? What if you loose all the data on it?

Better move the database’s dump somewhere else!

Amazon_S3Amazon S3 is a cloud storage service which ships with a very practical CLI tool.

In order to use it you need an amazon’s account and you need to install and configure the CLI tool.

Now that you have all the tools in place let’s see how we are going to automatically backup mysql databases to amazon s3:

  1. create a bucket on Amazon s3 (say my-backups)
  2. create a new folder inside that bucket (say my-backups/mysql)
  3. back up a database via mysqldump and store the dump into a folder in your server (say /var/local/backups/mysql/)
  4. synchronize that folder to Amazon s3’s my-backups/mysql

Steps 1 and 2 are done just the first time while 3 and 4 should be performed repeatedly (i.e. schedule them).

Here is a small bash script which does the job:


#!/bin/bash
#get the day of week as a number (i.e. 1 = Monday, 7 = Sunday)
DAY_OF_WEEK=$(date +%u)
#backup 'my_database' to the folder /var/local/backups/mysql/
#$DAY_OF_WEEK is appendend to the file's name in order to keep a weekly history of the backups
mysqldump -u user -ppassword my_database > /var/local/backups/mysql/my_database_$DAY_OF_WEEK.sql
#use aws (Amazon's tool) to sync your backups to Amazon S3
#/usr/local/bin/aws – absolute path to Amazon's CLI tool
#s3 – the service we want to use
#sync – the command to run
#/var/local/backups/mysql/ – local folder
#s3://my-backups/mysql/ – Amazon s3 folder
/usr/local/bin/aws s3 sync /var/local/backups/mysql/ s3://my-backups/mysql/

You can save this script into a .sh file, for example backup_mysql.sh, and then schedule every day in order to get an automatic backup. To do so, you need to add an entry to the crontab:

$ crontab -e

then add a line like the following at the bottom

00 00 * * * /bin/sh /path/to/backup_mysql.sh

this will run the script every day at midnight.

If you want to backup more than one database and keep backups separated you could add a parameter to the script:


#!/bin/bash
#get the day of week as a number (i.e. 1 = Monday, 7 = Sunday)
DAY_OF_WEEK=$(date +%u)
#backup the datababase passed as argument($1) to the folder /var/local/backups/mysql/
#$DAY_OF_WEEK is appendend o the file name in order to keep a weekly history of nthe backups
mysqldump -u user -ppassword $1 > /var/local/backups/mysql/$1/dump_$DAY_OF_WEEK.sql
#use aws (Amazon's tool) to sync your backups to Amazon S3
#/usr/local/bin/aws – the tool
#s3 – the service we want to use
#sync – the command to run
#/var/local/backups/mysql/ – local folder
#s3://my-backups/mysql/ – Amazon s3 folder
/usr/local/bin/aws s3 sync /var/local/backups/mysql/$1/ s3://my-backups/mysql/$1/

(pay attention ton the folders structure, it has to be consistent)

now you can pass the database name as a paramenter:

$ sh /path/to/backup_mysql.sh my_database

Testing query results with dbUnit

dbunit-logoWhen developing a data-warehouse you need to run tests on int to ensure that the datasets are consistent.

Call it data quality tests or just tests, they are needed if you want to trust your data.

Here is a simple tutorial on how to run test agains your dataset with dbUnit, by testing query results.

The idea is that you take a snapshot of a well tested resultset and the you test that the query which provides such dataset always return the expected result.

First of all you need a query to test, let’s say:

SELECT * FROM vw_test_me;

then we need to take a snapshot of it, to do so

Connection jdbcConnection = DriverManager.getConnection(<your db parameters>);
IDatabaseConnection connection = new DatabaseConnection(jdbcConnection);
// partial database export
QueryDataSet partialDataSet = new QueryDataSet(connection);
partialDataSet.addTable("test_snapshot", "SELECT * FROM vw_test_me");
FlatXmlDataSet.write(partialDataSet, new FileOutputStream(String.format("snapshot.xml","test_snapshot")));

Now we have an xml file called snapshot.xml which contains the resultset of the query above.

This is the starting point of our tests, everytime the query runs the result have to match with the resultset contained into snapshot.xml otherwise the test will fail.

dbUnit is a JUnit extension so the test should be written accordingly:

import junit.framework.TestCase;
import org.dbunit.IDatabaseTester;
import org.dbunit.JdbcDatabaseTester;
import org.junit.After;
import org.junit.Test;

public class TestQuery extends TestCase
{
    protected IDatabaseTester databaseTester;

    public TestQuery()
    {
        databaseTester = new JdbcDatabaseTester(<your db parameters>);
    }
    @Test
    public void testMyQuery()
    {
        ITable actualTable = databaseTester.getConnection().createQueryTable("test_query", "SELECT * FROM vw_test_me");
        IDataSet expectedDataSet = new FlatXmlDataSetBuilder().build(new File("snapshot.xml"));
        ITable expectedTable = expectedDataSet.getTable("test_snapshot");
        Assertion.assertEquals(expectedTable, actualTable);
    }
    @After
    @Override
    public void tearDown() throws Exception
    {
        databaseTester.getConnection().close();
    }
}

the testMyQuery method first runs the query we are testing and store its result into variable ‘actualTable’, then it retrieves the snapshoted resultset from snapshot.xml, eventually it checks if the two datasets matches.

Seguire le Olimpiadi di Sochi dall’Italia

Gli appassionati italiani di sport invernali non portanno seguire le Olimpiadi Invernali di Sochi sulla RAI. Se non hai SKY e non vuoi perdere neanche una gara ecco come seguire le Olimpiadi dal tuo computer:

  1. Crea un account sul sito http://www.unotelly.com/unodns/signup, cliccando su “try for free”
  2. Per vedere le olimpiadi in Inglese, (BBC2), selezionare United Kingdom nella sezione WDTV Owner
  3. Cliccare su ‘Continue to step 2’
  4. Selezionare il sistema operativo in utilizzo e seguire le istruzioni per impostare  UnoDNS
  5. Accedere al sito http://www.bbc.co.uk/iplayer/tv/bbc_two_england/watchlive

Il servizio sarà gratuito 8 giorni quindi non basterà per coprire l’intera durata delle olimpiadi, alla scadenza del periodo di prova occorrerà pagare secondo questa tabella http://www.unotelly.com/unodns/pricing.

Una volta finite le olimpiadi si consiglia di ripristinare le impostazioni del computer a meno che non si volgia continuare ad utilizzare il servizio UnoDNS.

Buona visione

P.S. per godere di una migliore visione collegate il vostro computer alla televisione.

WordPress Custom Type with Custom Post Field

Last days I’ve been developing a very simple custom post type for WordPress.

The aim was to allow the user to create an event, which in this case consists of:

  1. title
  2. description
  3. date

Title and description are already provided by the wordpress built-in post type, that means we just need to teach wordpress how to manage date.

In order to keep the code clean all the Custom Type’s code is inside a class Event which is inside a folder ‘custom_types’ whithin the theme’s folder.

/theme/custom_types/Event.php

class Event
 {
}

Then we add the class methods to WP on the functions.php after including the class itself:

    require(“custom_types/Event.php”);

First of all WP needs to know about our Post Type so that it has to be registered

 public static function create()
 {
 register_post_type('event',
 array
 (
 'description' => __('Events'),
 'labels' => array
 (
 'name'                    => __('Events'),
 'singular_name'            => __('Event'),
 'add_new'                => __('Add New'),
 'add_new_item'            => __('Add New Event'),
 'edit_item'                => __('Edit Event'),
 'new_item'                => __('New Event'),
 'view_item'                => __('View Event'),
 'search_items'            => __('Search Events'),
 'not_found'                => __('No events found'),
 'not_found_in_trash'    => __('No events found in Trash'),
 'parent_item_colon'        => ''
 ),
 'supports' => array('title', 'editor'),
 'public' => true,
 'capability_type' => 'post',
 'show_ui' => true,
 'publicly_queryable' => true,
 'hierarchical' => false,
 'menu_position' => null,
 'query_var' => true,
 'taxonomies' => array('post_tag', 'category'),
 'has_archive' => true,
 'rewrite' => array('slug' => 'events')
 )
 );
 }

we add this method to WP on the functions.php file via:

    //add custom post type to WP
add_action(‘init’, ‘Event::create’, 0);

Making the long story short register_post_type says to WordPress:

‘hey you, here is my custom type and it has the following features’

the meaning of all these features is well explained on the official documentation.

Now i’s time to add our columns

 

public static function columns($columns)
 {
 return array
 (
 'title'            => __('Title'),
 'event_date'    => __('Event Date'),
 'author'        => __('Author'),
 'tags'            => __('Tags')
 );
 }

 

Add to WP:

    //custom columns
add_filter(‘manage_edit-event_columns’ , ‘Event::columns’);

N.B. -event is the name of our custom post type

Here we are defining which columns will be used on the listing table of our custom type, the parameter $custom contains the default post columns, we could decide to unset some of them and add ours or (as shown above) just return an array with the columns we want.

As you can see we return 3 ‘built-in’ columns (title,author,tags) + our custom column (event_date).

WordPress also need to know what to output for each custom columns:

 

public static function backend_view($column)
 {
 global $post;
switch($column)
 {
 case 'event_date' :
 {
 echo(Event::get_date($post));
 break;
 }
 }
 }

Add to WP:

    //backend visualization
add_action(‘manage_posts_custom_column’, ‘Event::backend_view’);

here we say: ‘whenever you are rendering something about column event_date, print the Event’s date, we will se later on what’s inside Event::get_date($post), remember Event is the class containing our Custom Post type definition.

At this stage we are ready to find out a new menu point on the admin backend called ‘Events’, the table listing all the events (no one until now) and the page to edit/insert a new Event.

What? there is no ‘event date’ field on the edit page?
Yes, we need to specify a custom post field in order to be able to add this feature to our Events.

Adding the possibility of Managing custom columns

The previous steps don’t provide the feature on inserting/editing our custom column, to fix that point we need to introduce a custom post field:


public static function editor()
 {
 add_meta_box('event_date_box', __('Event Date'), 'Event::date_edit', 'event', 'normal', 'low');
 }
public static function date_edit()
 {
 global $post;
 $start = Event::get_date($post);
 echo("<label for='start_date'>Data:</label><input type='text' name='event_date' class='event_date' value='{$start}' />");
 }
public static function date_insert()
 {
 global $post;
if(!preg_match("/^(0[1-9]|[1-2][0-9]|3[0-1])/(0[1-9]|1[0-2])/([0-9]{4})$/", $_POST['event_date'], $pieces))
 return;
 else
 update_post_meta($post->ID, 'event_date', mktime(0,0,0,$pieces[2],$pieces[1],$pieces[3]));
 }
public static function date_delete()
 {
 global $post;
 delete_post_meta($post->ID, 'event_date');
 }

 

add to WP:

    //add custom fields to the editor
add_action(‘add_meta_boxes’, ‘Event::editor’);
//manage event_date
add_action(‘save_post’, ‘Event::date_insert’);
add_action(‘delete_post’, ‘Event::date_delete’);

add_meta_box add a new box on the post editing page, the third parameter is the one which specify which function will render the box.

As you can see Event::date_edit outputs an input field which our users will insert the date.

We need to define also what WP should do with the user input once the post is saved/deleted.

In the first case we store the input on the database by means of date_insert (actually here we check the date format and if it is valid we turn it into timestamp befora storing).

update_post_meta add/update a custom post field with name ‘event_date’ at the event we are saving.

The custom value is retrieved from the database and shown on he input if the alue itself exists, the methon by which we search for the value is the previously seen ‘Event::get_date’:

 

private static function get_date($post)
 {
 $data = get_post_custom_values('event_date', $post->ID);
 if(count($data))
 return date('d/m/Y',array_shift($data));
 else
 '';
 }

 

Here we search for the custom value on the db, if it exists we turn in back to a human-readable date otherwise just return an empty string.

In case we delete and event we teach wordpress how to delete the custom field ‘event_date’ by means of delete_post_meta.

Now we are done, our brand new Event Custom Type it’s on-line and users has the possibility of inserting the event date.

Anyway we could make the input a little bit more user-friendy with help of jquery…

Adding jQuery to our event date input

we need to include jquery and jqueryUI to the admin:

 

public static function add_js()
 {
 wp_enqueue_script('jqueryui', 'https://ajax.googleapis.com/ajax/libs/jqueryui/1.8.14/jquery-ui.min.js');
 wp_enqueue_script('jquery_i18n', 'http://ajax.googleapis.com/ajax/libs/jqueryui/1.8.14/i18n/jquery-ui-i18n.min.js');
 wp_enqueue_script('admin', WP_CONTENT_URL . '/themes/theme/js/admin.js', array('jquery','jqueryui','jquery_i18n'));
 }

 

jquery is already present on WordPress, thet’s why we pick just jqueryUI and all it’s translations from the Google CDN. Then we load our javascript (which is located on the js folder of our theme) setting as dependencies jquery, jqueryUI and its translations.

The same way we load the css styleshit:

 

public function add_css()
 {
 wp_enqueue_style('jqueryui', 'http://ajax.googleapis.com/ajax/libs/jqueryui/1.8.14/themes/flick/jquery-ui.css');
 }

 

Add to WP:

    //add javascript support
add_action(‘admin_print_scripts’, ‘Event::add_js’);
//add css style
add_action(‘admin_print_styles’, ‘Event::add_css’);

This way we make WP load all the scripts we need.

Finally we can write the few lines we need to properly add a calendar to our Event Date Input:

 

jQuery(document).ready(function()
 {
 jQuery('input.event_date').datepicker({dateFormat: 'dd/mm/yy'});
 jQuery('input.event_date').datepicker('option',jQuery.datepicker.regional["it"]);
 //close if opened on page load
 jQuery('#ui-datepicker-div:visible').hide();
 });

 

Here we go, now our custom post type admin interface in ready!

Complete Code

Event.php


<?php
class Event
 {
 public static function create()
 {
 register_post_type('event',
 array
 (
 'description' => __('Events'),
 'labels' => array
 (
 'name'                    => __('Events'),
 'singular_name'            => __('Event'),
 'add_new'                => __('Add New'),
 'add_new_item'            => __('Add New Event'),
 'edit_item'                => __('Edit Event'),
 'new_item'                => __('New Event'),
 'view_item'                => __('View Event'),
 'search_items'            => __('Search Events'),
 'not_found'                => __('No events found'),
 'not_found_in_trash'    => __('No events found in Trash'),
 'parent_item_colon'        => ''
 ),
 'supports' => array('title', 'editor'),
 'public' => true,
 'capability_type' => 'post',
 'show_ui' => true,
 'publicly_queryable' => true,
 'hierarchical' => false,
 'menu_position' => null,
 'query_var' => true,
 'taxonomies' => array('post_tag', 'category'),
 'has_archive' => true,
 'rewrite' => array('slug' => 'events')
 )
 );
 }
public static function columns($columns)
 {
 return array
 (
 'title'            => __('Title'),
 'event_date'    => __('Event Date'),
 'author'        => __('Author'),
 'tags'            => __('Tags')
 );
 }
public static function backend_view($column)
 {
 global $post;
switch($column)
 {
 case 'event_date' :
 {
 echo(Event::get_date($post));
 break;
 }
 }
 }
public static function editor()
 {
 add_meta_box('event_date_box', __('Event Date'), 'Event::date_edit', 'event', 'normal', 'low');
 }
public static function date_edit()
 {
 global $post;
 $start = Event::get_date($post);
 echo("<label for='start_date'>Data:</label><input type='text' name='event_date' class='event_date' value='{$start}' />");
 }
public static function date_insert()
 {
 global $post;
if(!preg_match("/^(0[1-9]|[1-2][0-9]|3[0-1])/(0[1-9]|1[0-2])/([0-9]{4})$/", $_POST['event_date'], $pieces))
 return;
 else
 update_post_meta($post->ID, 'event_date', mktime(0,0,0,$pieces[2],$pieces[1],$pieces[3]));
 }
public static function date_delete()
 {
 global $post;
 delete_post_meta($post->ID, 'event_date');
 }
public static function add_js()
 {
 wp_enqueue_script('jqueryui', 'https://ajax.googleapis.com/ajax/libs/jqueryui/1.8.14/jquery-ui.min.js');
 wp_enqueue_script('jquery_i18n', 'http://ajax.googleapis.com/ajax/libs/jqueryui/1.8.14/i18n/jquery-ui-i18n.min.js');
 wp_enqueue_script('admin', WP_CONTENT_URL . '/themes/theme/js/admin.js', array('jquery','jqueryui','jquery_i18n'));
 }
public function add_css()
 {
 wp_enqueue_style('jqueryui', 'http://ajax.googleapis.com/ajax/libs/jqueryui/1.8.14/themes/flick/jquery-ui.css');
 }
private static function get_date($post)
 {
 $data = get_post_custom_values('event_date', $post->ID);
 if(count($data))
 return date('d/m/Y',array_shift($data));
 else
 '';
 }
 }

 

functions.php


/*
 * CUSTOM POST TYPE: EVENTS
 */
 require("custom_types/Event.php");
 //add custom post type to WP
 add_action('init', 'Event::create', 0);
 //custom columns
 add_filter('manage_edit-event_columns' , 'Event::columns');
 //backend visualization
 add_action('manage_posts_custom_column', 'Event::backend_view');
 //add custom fields to the editor
 add_action('add_meta_boxes', 'Event::editor');
 //manage event_date
 add_action('save_post', 'Event::date_insert');
 add_action('delete_post', 'Event::date_delete');
 //add javascript support
 add_action('admin_print_scripts', 'Event::add_js');
 //add css style
 add_action('admin_print_styles', 'Event::add_css');

 

admin.js

 

jQuery(document).ready(function()
 {
 /*
 * datepicker
 */
 jQuery('input.event_date').datepicker({dateFormat: 'dd/mm/yy'});
 jQuery('input.event_date').datepicker('option',jQuery.datepicker.regional["it"]);
 //close if opened on page load
 jQuery('#ui-datepicker-div:visible').hide();
 });

 

Common Expenses: Lazy Loading

When developing the POJO classes for mapping the database I set the fetch type to LAZY. At first i didn’t pu much interes on it but then when I was actually fetching the data from DB I had to lern it better.

The target was to display all project’s data on a page, my first idea was to simply retrieve the project from db

[java]
(Project)session.get(Project.class, 1)
[/java]

and then display it:

[html]
<h3>${project.name}</h3>
<p>${project.description}</p>
<ul>
<c:forEach var="expense" items="${project.expenses}">
<li>${expense.name}</li>
</c:forEach>
</ul>
[/html]

But this didn’t worked. When an entity is instanciated all its collelctions are defined but the content is not loaded, infact lazy loading means Hibernate would fill the collenctions when them are requested.

If you try to retrieve all project’s expenses into the EJB there are no problems as an Hibernate session is opened. When we are into the view context (JSP in this case) there are no Hibernate sessions opened that’s why the expense collection could not be retrieved.

If the fetch mode is set to EAGER Hibernate load all entities collection whe the entity itself is istanciated. The view above would display all the data correctly but if we add a new expense it won’t be added to the collection as the main entity has already benn loaded.

The best solution would be create a Filter which open and close an Hibernate session at view context to lazy load the collections whenever are needed but seems like this way-around exists only in Spring Framework.

The fixeed this problem with in a ‘dirty’ way building a emporary container which collect the projects data every time the page is requested. This is not as performant as lazy loading because it fetches all the data at each request. But it finally make the view display all data.

Common Expenses: Stateful EJB, fail

Here we come with bad notes.

I was not able to implement some logic with Stateful Session Bean in my application. First it was difficult to find a place for such a thing inside my application then I wasn’t able to find good instruction on how to couple Stateful Bean with Servlet.

Common Expenses: Design Pattern

This post should have been one of the first actually.

Here is described the design pattern used on Common Expenses, I’ve used MVC as it is very straight forward: separate data management (model), application login (controller) and presentation (view).

As I post earlier the persisten layer is made with Hibernate, then the Businnes Login is implementend via Stateless EJB.

The application controller is a set of Servlet which get requests, work out them and send the result to the presentation layer.

The presentation layer is implemented with JSP.

Here is a schema of the whole application:

MVC
MVC

Common Expenses: EJB

To communicate with POJO entities I’m going to use EJB version 3.1.

As done before I will use annotation instead of a XML configuration file.

I will come back later with a full post on how to couple EJB with Hibernate.

Essentialy we need an Interface for make the bean accessible from other tools (in this case I’ll use Servelt to access EJB):

[java]
@Remote
public interface RegisterBeanRemote
[/java]

Remote means that this is he ‘server‘ class, in this instance our client will be the Servlets but if it would have been another java application we would have to distinguish between Remote and Local.

When the interface is ready we need a class to implement it:

[java]
@Stateless
public class RegisterBean implements RegisterBeanRemote
[/java]

Stateless means that the bean doesn’t keep a ‘memory‘.

There’s no need to use JNDI explixity because annotation will do it for us.