Recently in Development Category

PHP, Nagios and MySQL Replication

| 0 Comments
| | | |

Overview

MySQL replication is a handy way to distribue database processes across several servers. For example, a simple "master-slave" step up allows for a continuous backup of data from a primary database server, the master to a secondary backup server, the slave. But what if the slave server stops replicating for some reason? Not much of a good backup, if it fails to copy data for some undermined length of time.

The good news is that MySQL provides a simple, detailed query for checking if replication is taking place and will report errors, should they occur. The trick of course is getting notified when an issue does occur quickly. Given an existing Nagios setup for service monitoring at a PHP shop the only missing piece is some code.

The Details
First off, Nagios has the ability to supply arguments to a script as a script being invoked at the command-line. One common set of arguments for Nagios scripts are warning and critical thresholds. For example, a disk allocation script might take arguments to send a warning notification if the amount of free disk space reaches 20% and a critical notification if free space is 10% or less.

With MySQL replication one area of concern is the network. Any latency between the two servers can induce lag in synchronizing the slave server with the master server. Given this, why not pass along a threshold to our script setting checking how many seconds the secondary server is behind the primary.

For processing command line short form and long form options in PHP there is the getopt function:

        $shortopts  = "";
        $shortopts .= "w:"; // Required value for warning
        $shortopts .= "c:"; // Required value for critical

        $longopts  = array(
                // No long form options
        );

	// Parse our options with getopt
        $options = getopt( $shortopts, $longopts );

        // If slave is x second behind for warning state
        $delayWarn = $options['w'];

        // If slave is x second behind for a critical state
        $delayCritical = $options['c'];

Besides being in a critical or warning state, Nagios also has conditions for normal and unknown. Each state is associated with a status code that will be set upon completion of the script, hence the following associative array:

        // Nagios conditions we can be in
        $statuses = array( 'UNKNOWN' => '-1', 'OK' => '0', 'WARNING' => '1', 'CRITICAL' => '2' );

For the moment, we don't know what condition our replication setup is in. Nor do we have any additional information about the current state, so let's go ahead and define that as such:

        $state = 'UNKNOWN';
        $info = '';

The next step is to go ahead and connect to our slave MySQL instance and query its status using "SHOW SLAVE STATUS;"

		$db = new mysqli( $dbHost, $dbUser, $dbPasswd );

		// Prepare query statement & execute
		$query = $db->prepare( "SHOW SLAVE STATUS" )) {
		$query->execute();

The MySQL query is going to return a number of columns in a single result row. Of immediate concern is if the slave is in error state or not. For that we take a look at the columns labeled Slave_IO_Running, Slave_SQL_Running and Last_Errno.

        // If Slave_IO_Running OR Slave_SQL_Running are not Yes 
        // OR Last_Errno is not 0 we have a problem
        if (( $SlaveIORunning != 'Yes' ) OR ( $SlaveSQLRunning != 'Yes' ) 
        	OR ( $Last_Errno != '0' )) {

            	$state = 'CRITICAL';

If the slave server is not in error, then we'll go ahead and check how far behind it is, and set a warning or critical state given the earlier parameters from the beginning of the script:

        } else if (( $row['Slave_IO_Running'] == 'Yes' ) OR ( $row['Slave_SQL_Running'] == 'Yes' ) OR ( $row['Last_Errno'] == '0' )) {

        	// So far so, good, what about time delay, how behind is the slave database?
			if ( $row['Seconds_Behind_Master'] >= $delayCritical ) {

            	$state = 'CRITICAL';

            } else if ( $row['Seconds_Behind_Master'] >= $delayWarn ) {

            	$state = 'WARN';

            } else {

            	$state = 'OK';

            }

		}

Now that we have determined the state of the secondary database server, we can pass along some information for Nagios to process.

        // What to output?
        switch ( $state ) {

                case "UNKNOWN":
                        $info = 'Replication State: UNKNOWN';
                        break;

                case "OK":
                        $info = 'Replication State: OK Master Log File: ' .$MasterLogFile. ' Read Master Log Position: ' .$ReadMasterLogPos. ' Replication Delay (Seconds Behind Master): ' .$SecondsBehindMaster;
                        break;

                case "WARNING":
                        $info = 'Replication State: WARNING Master Log File: ' .$MasterLogFile. ' Read Master Log Position: ' .$ReadMasterLogPos. ' Replication Delay (Seconds Behind Master): ' .$SecondsBehindMaster;
                        break;

                case "CRITICAL":
                        $info = 'Replication State: CRITICAL Error: ' .$LastErrno. ': ' .$Last_Error. ' Replication Delay (Seconds Behind Master): ' .$SecondsBehindMaster;
                        break;

        }

All that is left is to transfer our information to Nagios via standard out and an exit code:

        // Need to set type to integer for exit() to handle the code properly
        $status = $statuses[$state];
        settype( $status, "integer" );

        fwrite( STDOUT, $info );
        exit( $status );

Putting it all together we get something like this:

#!/usr/bin/php
<?php

	$shortopts  = "";
	$shortopts .= "w:"; // Required value for warning
	$shortopts .= "c:"; // Required value for critical

	$longopts  = array( 
		// No long form options 
	);

	$options = getopt( $shortopts, $longopts );

	// If slave is x second behind, set state as warn
	$delayWarn = $options['w'];

	// If slave is x second behind, set state as critical
	$delayCritical = $options['c'];

	// Nagios conditions we can be in
	$statuses = array( 'UNKNOWN' =----> '-1', 'OK' => '0', 'WARNING' => '1', 'CRITICAL' => '2' );
	$state = 'UNKNOWN';
	$info = '';
	
	$dbUser = 'user';
	$dbPasswd = 'password';
	$dbHost = 'localhost';

	$db = new mysqli( $dbHost, $dbUser, $dbPasswd );

	if ( mysqli_connect_errno() ) {
	
		// Well this isn't good
		$state = 'CRITICAL';
		$info = 'Cannot connect to db server';

	} else {

		// Prepare query statement & execute
		if ( $query = $db->prepare( "SHOW SLAVE STATUS" )) {

			$query->execute();

			// Bind our result columns to variables
			$query->bind_result( $SlaveIOState, $MasterHost, $MasterUser, $MasterPort, $ConnectRetry, $MasterLogFile, $ReadMasterLogPos, $RelayLogFile, $RelayLogPos, $RelayMasterLogFile, $SlaveIORunning, $SlaveSQLRunning, $ReplicateDoDB, $ReplicateIgnoreDB, $ReplicateDoTable, $ReplicateIgnoreTable, $ReplicateWildDoTable, $ReplicateWildIgnoreTable, $LastErrno, $Last_Error, $SkipCounter, $ExecMasterLogPos, $RelayLogSpace, $UntilCondition, $UntilLogFile, $UntilLogPos, $MasterSSLAllowed, $MasterSSLCAFile, $MasterSSLCAPath, $MasterSSLCert, $MasterSSLCipher, $MasterSSLKey, $SecondsBehindMaster, $MasterSSLVerifyServerCert, $LastIOErrno, $LastIOError, $LastSQLErrno, $LastSQLError );

			// Go fetch
			$query->fetch();

			// Done
			$query->close();

			// and done
			$db->close();
	
			// If Slave_IO_Running OR Slave_SQL_Running are not Yes OR Last_Errno is not 0 we have a problem
			if (( $SlaveIORunning != 'Yes' ) OR ( $SlaveSQLRunning != 'Yes' ) OR ( $LastErrno != '0' )) {
		
				$state = 'CRITICAL';	
		
			} else if (( $SlaveIORunning == 'Yes' ) OR ( $SlaveSQLRunning == 'Yes' ) OR ( $LastErrno == '0' )) {
	
				// So far so, good, what about time delay, how behind is the slave database?
	
				if ( $SecondsBehindMaster >= $delayCritical ) {
				
					$state = 'CRITICAL';
				
				} else if ( $SecondsBehindMaster >= $delayWarn ) {
				
					$state = 'WARN';
				
				} else {
	
					$state = 'OK';
		
				}
			
			}
	
	
		} else {
			
			// Well this isn't good
			$state = 'CRITICAL';
			$info = 'Cannot query db server';			
			
		}
	
		// What to output?
		switch ( $state ) {

			case "UNKNOWN":
				$info = 'Replication State: UNKNOWN';
				break;

			case "OK":
				$info = 'Replication State: OK Master Log File: ' .$MasterLogFile. ' Read Master Log Position: ' .$ReadMasterLogPos. ' Replication Delay (Seconds Behind Master): ' .$SecondsBehindMaster;
				break;

			case "WARNING":
				$info = 'Replication State: WARNING Master Log File: ' .$MasterLogFile. ' Read Master Log Position: ' .$ReadMasterLogPos. ' Replication Delay (Seconds Behind Master): ' .$SecondsBehindMaster;
				break;

			case "CRITICAL":
				if ( $info == '' ) {
					
					$info = 'Replication State: CRITICAL Error: ' .$LastErrno. ': ' .$LastError. ' Replication Delay (Seconds Behind Master): ' .$SecondsBehindMaster;
			
				}
			break;
			
		}
	
	}

	// Need to set type to integer for exit to handle the exit code properly
	$status = $statuses[$state];
	settype( $status, "integer" );

	fwrite( STDOUT, $info );
	exit( $status );


?>

Chicago Open Data at Work

| 0 Comments
| | | |

A few years ago Blagica Bottigliero started the website Gals' Guide as an online forum for young women moving out on their own an into the "big city". Recently, I've been working with her on taking the site to the next step; building a web application utilizing the growing sets of data about life in Chicago.[1]

The Gals' Guide Map App is designed to combine different datasets about the city's various neighborhoods into one, assisting one in finding right place to live.

The web app is, somewhere between alpha and beta stages, not ready for general use or even rigorously browser tested, but ready for feedback. To that end, we've started showing the app to our various networks to gather feedback as it moves towards a general, full public release.

galsguide.png

The map and features therein have been influenced by other map mashups out there, such as the recent work done by the Chicago Tribune's News Applications team.[2]

Currently it incorporates data from the U.S. Census, the City of Chicago and Groupon. But, that's just the tip of the iceberg. There are plenty of other datasets about the city from sources such as county and state, Everyblock, Yelp, Grubhub and others.

Go, check it out and leave some feedback.



[1] This is also the next logical step for me from coding up PHP classes for the CTA's API and the City of Chicago's open data portal I started working on back in July.

[2] The team has a blog which includes a nice series of post on their work, I recommend taking a look.

Accessing Chicago, Cook and Illinois Open Data via PHP

| 2 Comments
| | | |

In Accessing the CTA's API with PHP I outlined a class file I created1 for querying the Chicago Transit Authority's three web-based application programming interfaces (APIs). However, that isn't the only open data development I've been working on recently, I've also been working on a class file for accessing the City of Chicago's Open Data Portal.

The City of Chicago's Open Data Portal is driven by a web application developed by Socrata. Socrata's platform provides a number of web-based API methods for retrieving published datasets2. class.windy.php provides the definition for a PHP object that wraps around Socrata's API providing PHP written apps access in turn to the city's open data.


Installation and Instantiation
The first step is to download the class.windy.php file from GitHub and save it in a location that the PHP application can access.

The next step is to include the file using the include function in the PHP application itself:

	// Load our class file and create our object.
	include_once( 'class.windy.php' );

Once the class file has been loaded, the next step is to instantiate the class:
	// Create an object to get city data
	$chicago = new windy( 'city' );

Note that initialization of of the new object requires providing the keyword 'city'. Since the City of Chicago, Cook County and the State of Illinois all currently use Socrata's platform for sharing government data, class.windy.php supports accessing all three data portals - more on accessing county and state data later.

Extra parameters can be provided during the construction of the object for additional customization. For example, by default, this class will request data in JSON form. But, Socrata does support other data formats3 so the PHP developer can have a bit of a choice as to how to handle and process the raw data, if desired.


Chicago Data
Let's say for example that one is looking for information about Chicago's roughly 200 neighborhoods. More specificity information about the rough boundaries around Chicago neighborhoods. Using the getViews4 method that class.windy.php provides, one can query for any dataset where the description might include the keywords 'neighborhood boundaries' and are tagged as KML:

	// Let's find any views that describe themselves as about 
	// Chicago's neighborhood boundaries and are tagged with 
	// KML data
	$views = $chicago->getViews( 
		'', '', 
		'neighborhood boundaries', 'kml', 
		'', 'false', '', '' 
	);
	
	echo "Here are the views with a description including 
	     'neighborhood boundaries' and are tagged as KML:\n";
	foreach ( $views as $view ) {
	
		echo "View ID: ".$view->id. " is named " 
		     .$view->name. " and is described as a " 
		     .$view->description. "\n\n";
		
	}

Depending on what datasets are available the output of this query may include the follow result:
View ID: buma-fjbv is named Boundaries - Neighborhoods - KML and is described as a KML file of neighborhood boundaries in Chicago. To view or use these files, special GIS software, such as Google Earth, is required.

Perhaps, using Google Maps, one wishes to create an interactive map of Chicago neighborhoods? Or better yet a closer look at the boundaries of a specific area in the city? Retrieval of the KML file can be done via the getFileByViewID function:
		// With our foreknowledge of datasets the file with 
		// view id buma-fjbv looks interesting, let's get 
		// that file
		if ( $view->id == 'buma-fjbv' ) {
					
			$file = $chicago->getFileByViewID( 
				$view->blobId, $view->id 
			);

			// Since KML is an XML notation for 
			// expressing geographic annotation, 
			// let's use SimpleXML to parse this data
			// Note: SimpleXML requires the libxml 
			// PHP extension
			$xml = simplexml_load_string( $file );

The above chunk of code nested in the previous example code's foreach loop compared the $view->id value of each dataset looking for a match to the specific KML file of Chicago neighborhoods. Once found, the getFileByViewID method, given the file id, found in $view->blobID and the known view id sends a query for and retrieval of the file from data portal.

Once the file has been loaded, the KML file, a type of XML format, can be parsed using the simplexml_load_string function which returns a SimpleXML object and can be traversed as needed:
			// Ok, now let's find out what the 
			// boundaries are for Albany Park
			foreach ( 
			     $xml->Document->Folder->Placemark 
				as $hood 
			) {
			
				if ( preg_match( 
					'/Albany Park/', 
					$hood->description 
				)) {
				
					echo "Here are 
						Albany Park's 
						boundaries: " 
						.$hood->
						MultiGeometry->
						Polygon->
						outerBoundaryIs->
						LinearRing->
						coordinates. "\n";
				
				}
			
			}

Thus the output of our search for the boundaries of Chicago's Albany Park neighborhood results in the the following geographic data points:
Here are Albany Park's boundaries:  
-87.724211,41.975689,0.000000
-87.724081,41.975576,0.000000 -87.723951,41.975578,0.000000
-87.723773,41.975580,0.000000 -87.723544,41.975583,0.000000
-87.723355,41.975585,0.000000 -87.722786,41.975592,0.000000
-87.721267,41.975605,0.000000 -87.720888,41.975608,0.000000
-87.720859,41.974088,0.000000 -87.720825,41.974085,0.000000
-87.720636,41.974088,0.000000 -87.720561,41.974079,0.000000
-87.720501,41.974061,0.000000 -87.720370,41.974010,0.000000
-87.720264,41.973981,0.000000 -87.720135,41.973962,0.000000
-87.719949,41.973890,0.000000 -87.719723,41.973819,0.000000
-87.719616,41.973775,0.000000 -87.719252,41.973625,0.000000
-87.719147,41.973578,0.000000 -87.719074,41.973535,0.000000
-87.718948,41.973461,0.000000 -87.718826,41.973396,0.000000
-87.718714,41.973346,0.000000 -87.718388,41.973226,0.000000
-87.718366,41.973218,0.000000 -87.718291,41.973195,0.000000
-87.718069,41.973097,0.000000 -87.717935,41.973023,0.000000
-87.717737,41.972915,0.000000 -87.717479,41.972781,0.000000
-87.717195,41.972637,0.000000 -87.717132,41.972604,0.000000
-87.716961,41.972517,0.000000 -87.716548,41.972280,0.000000
-87.716203,41.972086,0.000000 -87.715963,41.971978,0.000000
-87.715918,41.971968,0.000000 -87.715808,41.971941,0.000000
-87.715666,41.971921,0.000000 -87.715501,41.971920,0.000000
-87.715076,41.971932,0.000000 -87.714697,41.971930,0.000000
-87.714681,41.971930,0.000000 -87.714619,41.971930,0.000000
-87.714614,41.971930,0.000000 -87.714547,41.971937,0.000000
-87.714479,41.971942,0.000000 -87.714435,41.971949,0.000000
-87.714385,41.971952,0.000000 -87.714305,41.971954,0.000000
-87.714227,41.971960,0.000000 -87.714178,41.971965,0.000000
-87.714026,41.971969,0.000000 -87.713652,41.972033,0.000000
-87.713466,41.972078,0.000000 -87.713254,41.972129,0.000000
-87.712834,41.972277,0.000000  ...


Cook and Illinois Data Portals

As previously mentioned the Cook County and the State of Illinois also currently use Socrata's platform for sharing government data. For the moment that makes supporting city, county and state open data with one class file a trivial manner:

// Create an object to get county data
$cook = new windy( 'county' );

// Let's find any views that describe the boundaries of the 
// county forest preserves
$views = $cook->getViews( 
	'', '', '', 'county park boundaries', '', 'false', '', '' 
);
	
echo "Here are the views with a description including 
	'county park boundaries':\n";
foreach ( $views as $view ) {
	
	echo "View ID: ".$view->id. " is named " 
		.$view->name. " and is described as a " 
		.$view->description. "\n\n";
	
}


In Review
class.widy.php is a single PHP class file that primarily provides access to City of Chicago's data portal. The class implements functions for accessing all API methods and by default returns an object that a PHP developer can use to incorporate information about the City of Chicago into their PHP based application.

Since Cook County and the State of Illinois have also adopted Socrata's data platform, support for these additional data portals using Socrata's Open Data Platform has been included as a secondary feature of the class file.




1 And which can be found on GtHub

2 Known as SODA, a standards-based, RESTful application programming interface.

3 Such as XML, RDF, XLS and XLSX (Execl), CSV, TXT and PDF.

4 getViews is one of a collection of methods Socrata refers to as ViewsService. This collection of API calls provide for the retrieval and manipulation of datasets and metadata about datasets.   

Chicago's Technology Scene, circa 2011

| 0 Comments
| | | |

Roughly this time two years ago, I attended an entrepreneur and startup event in D.C. called Social Matchbox which I noted in a subsequent blog post of the same name.


While I know organizations and companies like those exists in many places I have yet to find a loose confederation of those individuals, organizations and companies similar to what I experienced in the Bay Area here in Chicago where I currently reside. I have however found such a network in Washington, DC and it is known as Social Matchbox.


Last year, I asked with all the things going for Chicago, high profile tech companies, top tier universities and a diverse population, "why don't tech people think of Chicago along the lines of a San Francisco, Seattle, New York, Boston or Austin?" Perhaps, I surmised, it was simply "that Midwestern work ethic, if we just work hard; the rewards and recognition will come on their own."

This year, I watched as the Chicago tech community rallied around TechWeek, a "celebration of a new technology epicenter unique among major world cities."

It's a start.

Alas, while I didn't get to attend any TechWeek specific events, someone has to keep an eye on the servers and write code for all these newfangled ideas, I did get a chance to meet up with a few other developers at the Chicago Open Data Hackathon. As a WTTW article wrote:


Chicago's city government has worked on developing its high-tech cred by initiatives such as publishing new city data sets online weekly and refreshing those sets nightly in order to increase the city's transparency ... On July 16, Google Chicago hosted a Chicago Open Data Hack Day. The event gathered 60 engineers, designers, and entrepreneurs to share ideas about using the City's open data to create new products and services."1


In my own participation of the hackathon, I did get a chance to create, what I hope will be, a useful PHP library (more on that later).

It's good to see Chicago get a little more boisterous about its tech creds. But it is also good to see Chicago go about about business as most Chicagoans do, as I did last week. As Orbit's Andy Crestodina notes "in a refreshing way, TechWeek was nothing new. Chicago has been doing this a long time and the tech community is an experienced crowd, many of whom have seen the boom and bust (and more booms and more busts) and lived to tell about it."

There is an end in all this technology means. We build apps to communicate. We open data to map relationships and piece out new meaning. The technology isn't an end unto itself. We work in technology to get something done.

And that's the innovation Chicago can bring to the table, even if it mostly goes unheralded.




1 I don't recall 60 people offhand, I probably would have put the count at around 30-40. But the hackathon was an all-day event and I know some people came and left, so, 60 total for the whole maybe true.

Where Software Methodology and Business Strategy Meet?

| 0 Comments
| | | |
Two Keyboards and a Mouse

Image by pdweinstein via Flickr

When I initially envisioned the article Web Development: Before and After the Client (local copy), my initial idea was to draw a line connecting Orbit Media Studio's overall business strategy, through the implementation of specific development methodologies, to the end result of the type of service Orbit offers.

Consider the description for Orbit's method of enhancing their content management system; "the focus is on breaking down the feature into workable steps and rapidly building them." This incremental process allows Orbit to keep coding solutions simple, quickly incorporate lessons learned from previous projects and sustain development of their codebase for the long term;

In doing so we consider what has worked for clients in the past along with growing trends such as social media integration.

But once a client enters the picture the goals change. A client has specific objectives, chief among them the desire for a stable website that is delivered on time and on budget. The change in goals requires a change in methodology;

In this sequential development process each step follows from the last. There is a specific beginning and ending. One step cannot be started until the previous step is completed and approved.

Thus, while these two methods have distinctly different goals unto themselves, Orbit uses both to bring about a specific end result in desired quality of results.

The point that I was trying to make got me wondering. Given a set of goals would most developers, or project managers, choose the development method(s) best suited to realizing those goals? Or would they choose the one they are most comfortable with?

Sure, most project managers or software engineers would recognize the different methodologies described above and that they focus on different objectives. But what's the most common way a development process is chosen?

Unfortunately, my experience over the years leads me to believe that most choose the one they know the best, is in fashion or is what is used "in-house." In fact many places I've interviewed at consider development methodologies like software platforms, they maybe an "Agile shop" just as much as they are a Mac or PHP shop.

This lack of connection between desired outcome and actual process haunts both business and technology managers in my opinion.

Then again, maybe I've had an odd experience with my career thus far?




Postscript: In perhaps an answer to my own question, while I was drafting this blog post, I came across this article, Design Driven Development. The post, from a Boston-based web design firm, outlines a similar organic evolution of their development process.

Chicago's Tech Scene: A Study in Contradiction?

| 0 Comments
| | | |
DSC01031

Image by pdweinstein via Flickr

One of the things that I have yet to get a handle on is the technology scene in Chicago. How can the country's third largest city, home to companies from Motorola to Threadless not have a thriving, engaging tech populous?  Why don't tech people think of Chicago along the lines of a San Francisco, Seattle, New York, Boston or Austin?

It's something I ponder once and awhile. Mostly I think we here in Chicago have an inferiority complex. Or maybe it has to do with that Midwestern work ethic, if we just work hard; the rewards and recognition will come on their own.  

I mean besides Motorola and Threadless, I can think of a number of established tech companies such as Abbott Laboratories, Tellabs and NAVTEQ. Or trendy web companies such as Orbiz, 37signals, Groupon and Grub Hub. And that's not including non-Chicago companies that have some sort of technical presence in the area, such as Google, BP or AT&T. Or the countless under-the-radar companies such as Orbit Media Studios.

Mix in top-tier schools in the region such as; Northwestern University, University of Illinois at Urbana-Champaign and the University of Wisconsin at Madison as well as federally funded research institutions Argonne National Labs and Fermilab and well, I think you start to see my point.

Thankfully, I'm not the only one who ponders why Chicago isn't seen as a tech city.

Insight Labs is leading a discussion in connection with midVenturesLAUNCH called "Not Being Silicon Valley" in which they ask questions such as "If Chicago were the new center of the technology world, what would it look like?" "What assets and advantages could we leverage to make Chicago the dominant technology powerhouse?" "What changes must occur to make the Second City number one for tech startups?"

Naturally my own answer jumps off with my thought that Chicago isn't seen as a tech-centric place to be, that there is no tech scene or as I commented, "I 'knew' going in that if I really wanted to make a career as a programmer I would end up working in Silicon Valley. Eventually I returned to Chicago (the Bay Area is damned expensive, even for a well-paid programmer) but even now I wonder if I'm missing on new opportunities by staying in Chicago."

More to the point of answering the question at hand, what changes must occur? I say understanding that we already have healthy technology assets, that opportunities do exist and a change of outlook is needed "to do a better job of recruiting and mentoring" in building a vibrant tech community.

What do you think?

Submitting a Perl module to CPAN

| 3 Comments
| | | |
O'Reilly created Programming Republic of Perl ...

Image via Wikipedia

Last May, I finally made available for download the first version of Webservice::Viddler on CPAN (Comprehensive Perl Archive Network) after having written about it in March. One of the reasons for the delay in making the packaged source code available for download had to do with one simple fact, the Viddler module was my first ever submitted for public consumption.


Now, don't misunderstand me, Webservice::Viddler is not the first Perl module I have ever written. I've been writing code in Perl for about as long as CPAN has existed. However, as is the case with my current work in PHP for Orbit Media Studios, most of the Perl code I wrote over the years was on behalf of a employer of some sort. As such the ownership of the code, and the right to distribute, rested with them, not me.


So then, why the delay? Well there are a couple of reasons. First of all, the code I posted in March was a proof of concept based on some work I did for Orbit at the time. While the basic framework of the module worked the "proto-module" didn't implement all of the functionally provided by the Viddler Web API.


Secondly, I needed to organize the source code for proper distribution on CPAN as well as get the packaged distribution uploaded and made available.


The Packaging

Before I did anything, I wanted to make sure that my module had all of the necessary files. In doing a little googling, I came across a blog post entitled Submitting a CPAN module which outlines the basic steps:


  1. Apply for an account on PAUSE (Perl Authors Upload Server)

  2. Organize your code

  3. Profit

Not too complex, granted, but organized how?


As the author notes there doesn't seem to be any "what the package must have" rules. However, as anyone who has worked with third-party Perl modules knows, there is an accepted process and organization of code that all modules tend to adhere to.


After a little more googling I came across module-starter, a handy command-line interface to a Perl module, Module::Starter, which does all the work of creating a base module for distribution. After adding in my code and documentation I quickly had something close to ready.


Close, but not complete. Besides making the package useful to install, I wanted to make the code useful to modify. For that I turned to perltidy a Perl script which indents and reformats Perl code to makes it easier to read and follow.


Great! Now here is where this can get interesting (and where a lot of suggestions, if not outright rules, do exist). If you follow the steps in the order above, good, because that means while waiting for a PAUSE account to come online1, proper considerations can be made for the naming of the module.


The Namespace

The PAUSE documentation On The Naming of Modules notes that "a module name must accomplish quite a bit in a few characters", such as provide context as to what the module does or problem it addresses. Also of importance is the fact that "once chosen, you rarely have the opportunity to change it after people start using it."


So a little careful consideration is in order.


Also, it is important to note that namespaces, by definition, are unique. Besides providing a singular meaning, the name cannot be shared with some other module, public or otherwise. As such it is recommend that PAUSE developers register the namespace of the module written. Again, this isn't a hard and fast rule per se, but a recommendation to avoid duplication and improve searchability.


That about covers the basics. Now, if you will excuse me, I need to take some time to address these test results.




1 The PAUSE documentation suggests allowing up to "three weeks for proceeding" but that requests a usually completed "within a week."

Amazon Looking to Add Touchscreen to Kindle?

| 0 Comments
| | | |

First published: 3rd of February 2010 for Technorati


The New York Times is reporting that Amazon has acquired Touchco, a small New York-based start-up specializing in touchscreen technology and is planning to incorporate the company's six employees and technology into its Kindle hardware division, named Lab126, which is located in California.

Unlike Apple's touchscreen technology which has a limited number of touch points, Touchco's solution is reported to detect an unlimited amount of simultaneous touch points - which provides greater accuracy - as well as distinguish between different levels of pressure, at a cheaper cost.

Of course last week Apple announced their latest creation: the iPad, a 10-inch tablet-style multi-touch device which will also include an new app called iBooks, a virtual bookshelf containing the user's personal collection of electronic books that can be purchased and read in a manner similar to music on an iPod. "If you've used iTunes or the App Store, you're already familiar with this," Jobs said.

Currently, Amazon's Kindle, unlike Apple's iPad - which uses a more traditional technology to display colorful, interactive content - uses a different display technology known as E Ink.

E Ink is a specific proprietary type of electronic paper manufactured by the company of the same name. While the principle advantage of E Ink is in its ultra-low power consumption and flexibility, it is only commercially available with a small color range (grayscale) and responsiveness, limiting the Kindle's content to mostly static content, such as novels.

However, Fast Company, in reporting E Ink's merger with a much larger business partner last year, noted that the acquisition was designed "at speeding the development of color e-paper." At the time Amazon CEO Jeff Bezos was quoted as saying that a color Kindle was "multiple years" away, but Fast Company went on note that E Ink's Vice President Sriram Peruvemba suggested that color displays would begin being available at the end of 2010.

Of course Amazon has already moved to expand the appeal of its eReader, as last month it announced plans for an application store in hopes of getting outside developers to create the same array of programs that are now available for the iPhone and iPad.

Given the Apple and Google's "pole positions" in the hardware and software development of mobile technologies, could Amazon's Kindle be a dark-horse contender for our lazy Sunday afternoons?

Google's Chrome OS in 2010

| 0 Comments
| | | |

First published: 20th of November 2009 for Technorati

Yesterday Google hosted a small technical introduction to its new Chrome Operating System (OS), which is scheduled for release on new netbooks by the end of 2010.

Google's vision for Chrome is to build on the concept of the Web as a ubiquitous computing platform. As outlined by Sundar Pichai, Google's Vice President of Product Management, "in Chrome OS, every application is a web application." Which means at the heart of everything is Google's Chrome Web Browser, modified to run all on its own, "It's just a browser with a few modifications. And all data in Chrome OS is in the cloud."

That in turn allows Google to provide a quick, nimble system that can "be blazingly fast, basically instant-on." As demonstrated on the test system, built on a modified Linux kernel, went from power-on to surfing the Web in 10 seconds.

In essence, Google Chrome is a cross between Google's cellphone software Android, which is also hosted on the Linux kernel, and the Chrome browser. However, unlike Android, which is built on a modified Java platform for third-party applications to be built and run on, Chrome OS is built to run today's rich web applications built on AJAX as well as tomorrow's web applications built around the draft HTML5 standard.

But what does this mean for Microsoft and Apple? While Google's development of their own operating system is indeed a direct challenge to Microsoft's bread and butter family of Windows, Chrome OS isn't better than Microsoft Windows product or Apple's Mac OS X. Nor is Google's OS even focusing on the traditional tasks of managing the interface between the local hardware and user.

Instead, Google's operating system is about simplifying and enhancing access to applications online. Not so much a replacement of current personal computers, but an alternative to getting online and accessing applications such as Google Docs or Twitter.

Anything that can be done on any standard Web browser on Windows, Mac and Linux can be done on Chrome which means Google's soon-to-be operating system is designed to leverage the growing collection of service-oriented software that can be found online, including, of course, Google's own suite of applications.

The trick for Google now is not just in implementation, but also adoption. Building on the growing trend of netbooks helps, but network computing itself is hardly a new concept.

Adding Government to the Social Web

| 0 Comments
| | | |

First published: 13th of October 2009 for Technorati

In part because of Barack Obama's success in leveraging the Internet during his Presidential Campaign and in part because of the general success of "Web 2.0 companies", there has been an explosion of discussions on how one can leverage the growing sophistication of our social interactions online.

Consider it: we cannot go a few hours these days without hearing about individuals, organizations and businesses experimenting with how to put some social networking site to work for them. Everyone wants to engage individuals with cool websites and apps - and make money doing it.

This growing complexity of the Web as a computing platform over the last half a dozen years or so has had to do with the ever increasing access to data repositories online, the development of social applications that access that data and the near ubiquitous Internet access many of us find ourselves with these days. In turn, we get businesses, organizations and individuals clamoring for tools and knowledge on how best to leverage this latest version of the Web.

But not all of the discussions about the Web these days revolves around making money. What about government? If the bold new social web can engage people in politicking, why can't that same tool be used in politics?

That is if Obama can leverage our social interactions to raise funds and engage like-minded individuals, why can't the government - at any level - use those same tools to engage its citizenry and help foster debate and execution of government policy? After all this version of the web is about social networks, social engagement and social collaboration, it's the perfect tool for the debate and analysis of policy both successful and failed?

Enter "Government 2.0": a growing trend that intersects politics, government and technology, bring democratic governments to their people, online. While still in its infancy, the idea has been gaining traction all over the world but most notably in the United States, Canada and the United Kingdom.

For example, in the United States, President Obama's Chief Information Officer for the Executive Branch of the Federal Government recently launched a new website, Data.gov, with the specific purpose of increasing "public access to high value, machine readable datasets" which is a critical step to bring about the creation and utilization of online political engagement.

Critical, but only a first step. Data from other parts of the government, such as local and state run institutions, are still in need of exposure. Those application developers who are just starting to incorporate these new repositories of open data into their social applications will be able to expose important and interesting information to the public at large.

A trend to keep an eye on, even if you're not a technologist. As Thomas Jefferson said in his First Inaugural Address, "The diffusion of information and the arraignment of all abuses at the bar of public reason, I deem the essential principles of our government."

About the Author

Paul is a technologist and all around nice guy for technology oriented organizations and parties. Besides maintaining this blog and website you can follow Paul's particular pontifications on the Life Universe and Everything on Twitter.

   
   


Subscribe:
Add to Google Reader or Homepage
Add to My AOL

Add to netvibes
Subscribe in Bloglines
Add to Technorati Favorites

Powered
CentOS
Apache
MySQL
Perl
Movable Type Pro