John Napiorkowski’s Perl Development Blog

A few months ago, when discussing how to support the 'darkpan' (what Perl programmers call the vast body of perl code floating around the internet or in private companies that we know nothing about) I suggested leveraging the vast CPAN distribution system as a sort of continuous integration/build service which we could offer as a for pay service to companies using Perl.  This would have the dual benefit of encouraging corporations to package up their perl code properly (ie as  CPAN module with a Makefile.PL, etc) and could help raise money to sponser Perl development.

What I'm thinking about would be similar to a ruby only service I noticed: http://runcoderun.com

Check that out and let me know what you think.  What kind of services would the 'quasipan' be required to offer in order to entice companies into paying for it?  For example, what do you think of the idea of being able to build a EC2 or other cloud image type from a base OS that intergrate any given set of CPAN and Darkcpan modules?  Like, "I want a virtualbox VM with Perl 5.10.1 on debian with git, postgresql and Catalyst preinstalled, although with my private repo of proprietary code?"

Well Perl Ironman Readers,

Based on the public and private comments from the previous blog I decided to go ahead and write something (I hope) cpan worthy.  It should show up shortly over here but until then you can check it out on github (or clone / offer improvements)

I'm not in love with the name.  If you have a better idea please speak up, but I don't like to let naming issues stop  me from publishing.

Thanks!

I find as part of my normal course of writing perl moose based code, I do the following quite often:

Basically I tend to use the above with a core application class that is delegating work to several other classes.  In the above case, my general Library Class contains a Album class (presumable your library would hold an album and perhaps several other things).  In my Application logic I'm going to be calling methods on Album which have been properly normalized to the class which attempt to encapsulate the problem domain it attempts to solve.

I find the above form, although a bit verbose, to give me a lot of flexibility, particularly when I bring the application model into Catalyst via something like Catalyst::Model::Adaptor, which let's me define all the args in the global catalyst configuration.  Also, for testing, it's nice to be able to override the default classes, swapping in say a Mock Object or similar for the purposes of test cases.

In general use this construct commonly to avoid hardcoding all the connections between the elements of my applications.

Usage might be like:

Although I as mentioned, typically I instantiate this via Catalyst or as part of a more complicated application using an inversion of control container like Bread::Board. 

One of the things that I personally find exciting about working in IT is how quickly the field grows and changes. Seemingly overnight vast bodies of knowledge and hard earned abilities can be supplanted. Five years ago, the job that took half the day and the coordination of a group of skilled workers can today be churned out by an intern using some free tool like Google Analytics.

The biggest downside to this constant leveling trend is how often I, as a technology professional, am challenged to prove the value of my work, my salary and my knowledge. Thus the crisis we, as professionals in this field, face. It is a crisis on two fronts. The first is the relentless loss of value to the skills we worked hard to master due to shifts in technology. The second is in the fact that most workers today more or less have to learn some degree of technology expertise in order to do their job well, and the best and brightest of our colleagues can often accumulate significant skills.

This challenge is brought to us in many ways, but often it is brought to us when we are least expecting it and am not really ready to answer it. Sometimes its appropriate, as during a job interview, or when reviewing or auditing code. There is definitely a right time to challenge your practices, least you fall into a dangerous rut. On the other hand, I'd like to reach out to my colleagues across the various departmental boundaries and point out it can be unproductive to start challenging technology decisions in an inappropriate forum. We all have a stake in finishing a project and a domain of action for accomplishing that. I would never start questioning legal matters at a direct reports meeting, and I ask that you refrain from speculating that this all could have been more easily done using VBA and Excel, just because you heard so from a friend of a friend.

This is why I would wish that more companies would break down the jealously guarded walls between IT and the rest of the company. Technology meetings should never be limited to the workers in a given department. In return, for opening the door, I ask that my non IT colleagues try to show some sensitivity and realize that often your off the cuff suggestions can sound to me like you are saying my skill and ability are no longer needed. Often what I hear is that I'm going to be forced to work with yet another ill conceived technology directive. For this sensitivity is needed all around.

So, this is a Perl blog and you are wondering what's the Perl angle. Given the diffusion of technical ability, from technology guru down to mail room working, I believe that Perl is an ideal language for sharing at all these levels. Perl is ideally suited for most problem domains and most levels of technical ability. Perl has been used by guru's to build sophisticated websites, used by system administrators to smooth the daily troubles, and used by neophytes to bang out a quick CGI front-end to a local database. Its a language comfortable with the ambiguity and with the interdisciplinary nature of IT as it exists in the wild today. Perl Programming embraces the world as it is. As such it's an ideal choice and nearly always my first choice.

When I first started using Perl Catalyst for web application development I felt appalled by how sloppy it seemed to pass important application data around in a secondary global namespace, the Catalyst Stash.  I guess it's context global, but it stills seems messy to me in that it seems to poorly replicate features already built into Perl, and for which Perl can help you use correctly.  I mean, a global is bad enough, but if you use strict and warnings you will at least get warned if you abuse a global, such as if you try to use it before it's initialized and all that.  But the Stash is basically a Hash, and it's prone to typo issues, etc., basically all the evil stuff that I tried to banish from my Perl applications years ago.

It's one of those things that seems fundamentally busted to me, yet it works well enough for simple to medium cases.  It just feels to me like it decouples control way too much, and in my experience the stash becomes a sort of no man's land.

I think it's one of the three things that bother me about Catalyst (with the ->forward/go/visit/detach versus method call ambiguity and the way the default TT view guesses a template based on the action name being the other two things that work acceptably enough but still manage to bother me quite a bit).

Thoughts?  Am I being too controlling here?  Should I just go with the flow or are there others out there that are also bothered by this?

The latest Perl localizes very nicely.  Following these very simple instructions I was able to install it into my home directory.  I then modified my .bash_profile to search the local perl bin area:

[jnapiorkowski@perlbuilder ~]$ cat .bash_profile
# .bash_profile

# Get the aliases and functions
if [ -f ~/.bashrc ]; then
        . ~/.bashrc
fi

# User specific environment and startup programs
PATH=$HOME/bin:$HOME/local/bin:$PATH

In this case I added "$HOME/loca/bin" so that search path lookups find my Perl over the system Perl.  I then used "http://github.com/jjn1056/catalyst-app-example-locallibapp/tree" to setup my fully local::lib deployments of my various catalyst applications.  Everything worked as expected.

Result?  Localized Perl that is separate from you system Perl, plus local::lib for each deployed application, keeping everything nice and cleanly separated.  Now when I need to deploy this to stage and then to production I can simple use rsync against my home directory and target my stage or production boxes (since my build box is a template for stage and production).  Combined with CPAN::mini I could even have my own local cpan server with 'approved' versions of modules.  Personally since I try to help fix stuff when it breaks, I try to grab the newest of everything, but if you are on a more conservative setup, looking at your own minicpan might be valuable.

I know this has been reported elsewhere, but the very excellent, "Higher Order Perl", is available as a free download.

This is definitely worth buying if you can spare the cash, or can convince your boss to pay for it.  However if you can't, or if you just really want something you can read on your notebook or mobile phone while riding the train, this pdf is very high quality.

If you haven't heard about this book, it's a fantastic exploration of how powerful and beautiful Perl syntax can be.  I know people often charge that Perl is 'read only' or 'line noise'.  Taking a quick look here should silence that criticism.

If you are trying to learn Perl and looking for other free books, check this page out.  Or, check the general Learning Perl page.  Or just ask me what to do in the comments section below :)

One of my personal goals for this year is to reduce some of the boilerplate code I have to write in order to get working with my applications.  This effort to ease Perl Programming is inspired by great projects like Moose, which really allows me to get focused on the job at hand and not worry about framework and setup.  One area that I'd like to see improved relates to the effort involved in setting up development and testing databases.  My recent cpan release of Test::DBIx::Class is my swing at the latter; this blog is my outline and RFC for the former.

One of my goals for Test::DBIx::Class is to make it trivial to deploy, seed and cleanup testing databases.  Currently we have the ability to automatically deploy DBIC based Schemas to SQLite, MySQL and Postgresql.  This ability is built on top of DBICs deployment code as well as some other bits from CPAN.  This makes the job of deploying, setting up and testing as easy as:
    
    use Test::More;
    use Test::DBIx::Class qw(People);

    fixtures_ok 'basic',
        'basic fixtures installed';

    is_fields 'first_name', People, [qw/john vanessa vincent/],
        'Got expected first names';

    done_testing();

Which will automatically create a database (using SQLite by default, but you can override and deploy to MySql or Postgresql as well), install the 'basic' fixtures (from a configuration file, but you can 'inline' create statements easily as well, see the docs for more) and test the schema resultset called 'People' to see if that set contains the asked for first_names.  This reduces a lot of boilerplate code for deploying the database, checking it, etc (and we've all written 10 half baked version of that, right :) ) and offers some helpers for actually testing the data.  For example, the 'is_fields' will ignore sorting order by default, comparing just the actual set.  The above test assertion would probably have to be rewritten as:

    is_deeply [sort map { $_->first_name } $schema->resultset('People')->all],
        [qw/john vanessa vincent/],
        'Got expected first names';

If you wanted to duplicate most of it's functionality, but there might be edge cases missed, like date fields and columns that inflate, weird differences in null handling, etc.  And that's for a very simple case where you are testing a single field.  If your test is more involved the syntax becomes increasingly verbose.

One of the things I really wanted for this module is to make it very easy to create a test database not only for SQLite, but for at least Postgresql and Mysql.  Both these goals are helped along by the cpan distributions Test::mysqld and Test::postgresql which autodetect the presence of Mysql or Postgresql (they need to be installed, but don't need to be running) and creates a temporary database installation in /tmp or the directory of your choice.  These databases live until the final tests complete and are then automatically cleaned up, unless you want to keep them, using a configuration setting or an %ENV variable.

This grants you a lot of benefits when you are trying to speed development along.  You don't need to login and create the testing database, worry about setting up all the correct permissions, etc.  If you are working in a multi developer company (like most of us) you probably have to create testing databases for everyone, unless you are willing to deal with the chaos of everyone's tests banging into each other.  This method basically just creates a database for the programmer on the fly, reducing that effort considerable.  Finally, since each temporary database is given it's own area to live, this allows you to run prove in parallel using the --jobs (or -j) option.  Usually you can't run database tests in parallel, since each test will essentially be contending for a single testing database instance, using classic methods.  Running parallel test jobs can greatly speed completing your tests, even with the overhead needed to instantiate a test database.

Now that I feel pretty comfortable with the code functionality, I am looking to factor out some core bits and use it to build a similar tool for development databases.  This would be a trait loadable by newer versions of Catalyst::Model::DBIC::Schema that would autodetect your target database, instantiate and deploy one as needed.  This way it would make it much easier to just get going with development, particularly if you are following a methodology were each developer has a personal development database.

I'd like your feedback on the idea, and request comments on the features you'd like to see most.  For example, I am currently working on supporting Replication in Test::DBIx::Class, since if you are using replication in production then your test cases really should run against a replicated setup.  This may not be needed in development.  On the other hand we might want our development databases to support the versioning system built into DBIC.  Since in test I generally build the database from scratch each time, I don't care about versioning so much.

My imagination tells me we'd want something called  “Catalyst::TraitFor::Model::DBIC::Schema::AutoDeploy” which would add a configuration option to your Catalyst::Model::DBIC::Schema called 'target_driver' or similar so you could specify SQLite, MySQL or Postgresql, as well as whether you wanted to preserve the database between running the application server and so forth.  I also figure by default to built the database files in $home/share/data, or similar, (again looking for suggestions on the best default).

This would be intended primarily for development, but would probably also be useful for people deploying catalyst applications to shared hosting setups.  This would allow one to basically deploy everything for an application to a single directory root.

Thoughts?

FUSE is an excellent system for mounting various types of storage devices to the filesystem in user runmode time.  Now, Perl Programming makes it trivial to create and mount new storage types using this framework.

http://search.cpan.org/dist/MooseX-Runnable-Fuse/

This is a MooseX::Declare friendly toolkit for building custom FUSE modules, and functions as an ease of use wrapper on top of the comprehesive FUSE CPAN distribution.

Props to all the involved authors!

There's a new version of URI on CPAN.

This hopefully fixed bug number 2 of my project to solve 10 Catalyst related installation errors by the end of the Northern Hemisphere summer.  Just 8 more to go between now and Sept 22, 2009!

I'll be working on Mac OSX related installation issues for the next item.  I'd like for Perl Programming on the Mac to be much smoother than it is now.  It really is an easy target since the Mac underneath can install a pretty generic unix toolchain and userland, just a few rough edges causing trouble.

UPDATE:  Fixed broken link in post as reported.  Thanks!