33 posts tagged “perl”

A few months ago, when discussing how to support the 'darkpan' (what Perl programmers call the vast body of perl code floating around the internet or in private companies that we know nothing about) I suggested leveraging the vast CPAN distribution system as a sort of continuous integration/build service which we could offer as a for pay service to companies using Perl.  This would have the dual benefit of encouraging corporations to package up their perl code properly (ie as  CPAN module with a Makefile.PL, etc) and could help raise money to sponser Perl development.

What I'm thinking about would be similar to a ruby only service I noticed: http://runcoderun.com

Check that out and let me know what you think.  What kind of services would the 'quasipan' be required to offer in order to entice companies into paying for it?  For example, what do you think of the idea of being able to build a EC2 or other cloud image type from a base OS that intergrate any given set of CPAN and Darkcpan modules?  Like, "I want a virtualbox VM with Perl 5.10.1 on debian with git, postgresql and Catalyst preinstalled, although with my private repo of proprietary code?"

Well Perl Ironman Readers,

Based on the public and private comments from the previous blog I decided to go ahead and write something (I hope) cpan worthy.  It should show up shortly over here but until then you can check it out on github (or clone / offer improvements)

I'm not in love with the name.  If you have a better idea please speak up, but I don't like to let naming issues stop  me from publishing.

Thanks!

I find as part of my normal course of writing perl moose based code, I do the following quite often:

Basically I tend to use the above with a core application class that is delegating work to several other classes.  In the above case, my general Library Class contains a Album class (presumable your library would hold an album and perhaps several other things).  In my Application logic I'm going to be calling methods on Album which have been properly normalized to the class which attempt to encapsulate the problem domain it attempts to solve.

I find the above form, although a bit verbose, to give me a lot of flexibility, particularly when I bring the application model into Catalyst via something like Catalyst::Model::Adaptor, which let's me define all the args in the global catalyst configuration.  Also, for testing, it's nice to be able to override the default classes, swapping in say a Mock Object or similar for the purposes of test cases.

In general use this construct commonly to avoid hardcoding all the connections between the elements of my applications.

Usage might be like:

Although I as mentioned, typically I instantiate this via Catalyst or as part of a more complicated application using an inversion of control container like Bread::Board. 

When I first started using Perl Catalyst for web application development I felt appalled by how sloppy it seemed to pass important application data around in a secondary global namespace, the Catalyst Stash.  I guess it's context global, but it stills seems messy to me in that it seems to poorly replicate features already built into Perl, and for which Perl can help you use correctly.  I mean, a global is bad enough, but if you use strict and warnings you will at least get warned if you abuse a global, such as if you try to use it before it's initialized and all that.  But the Stash is basically a Hash, and it's prone to typo issues, etc., basically all the evil stuff that I tried to banish from my Perl applications years ago.

It's one of those things that seems fundamentally busted to me, yet it works well enough for simple to medium cases.  It just feels to me like it decouples control way too much, and in my experience the stash becomes a sort of no man's land.

I think it's one of the three things that bother me about Catalyst (with the ->forward/go/visit/detach versus method call ambiguity and the way the default TT view guesses a template based on the action name being the other two things that work acceptably enough but still manage to bother me quite a bit).

Thoughts?  Am I being too controlling here?  Should I just go with the flow or are there others out there that are also bothered by this?

I wanted to drop in and explain my lengthy absence, especially given some interesting "part two" articles in my queue that several of you have emailed me hoping for an update.  In particular there's a pending summary of all the feedback regarding "Why All the Hate", the long overdue "Blue Child" installment (regarding end to end development of a Perl Catalyst application) and on a lighter note, my decision regarding "Is it time to rotate the Profile Pic?"  There's a few other items in my PENDING IRONMAN BLOGS directory as well.

Earlier this summer I was diagnosed with an early form or tendonitis, and I've been struggling to avoid surgery to repair it.  That has limited my typing time substantially and although I've been using voice recognition to help pick up the slack, I can't use it on my train rides to and from work (it annoys the other passengers too much, similarly to what a loud mobile phone conversation might provoke).  So I guess I'm out of the Ironman running, although I am cool with that since so many of you are stepping up to the job.  I also recruited a new blogger, a Perl newbie, who is trying to record his learning Catalyst adventures.

Anyway, I'm not here to bitch about all that.  I am really annoyed by the following issue regarding database design and that's what I am going to bitch about.  My monumental annoyance is as follows:

Lookup tables.

I'm not talked about generic lookup tables (see this article for what's bad about those.)  I'm talking about the kind most of us use all the time.  For example, you have the logic "Each Address must have a State".  As part of normalizing, you create Address and State tables, with a foreign key constraint between Address and State.

This seems natural but always gives me a queasy feeling.  To me, they seem to be blending the boundary between schema definition and constraints with actual row data.  This makes it hard without using triggers to have special constraints on certain types of rows in the look up.  For example, California has a lot of special rules regarding pretty much everything, which results in tons of evil code like:

    if($row->name eq 'CA') { ## do something special }

Careful design in your ORM can mitigate the issue.  But I still don't like what seems to be a very hard solution to a problem we all must have several times a day.  

I also don't like how in look up tables, the row data has to be part of the database design.  DBIC makes it easy to deploy a DDL of a database based on the classes you define, but then you still need to go in and manually add a bunch of "insert into gender(name) values('female')" to the generated DDL file.  This basically takes a nice and neat automated system and makes it manual again.

Plus right now we can't use DBIC Versioning to manage the differences in this row data.  DBIC Versioning only handles DDL differences, which is as expected since you would imagine your actual data to be data, not logic or constraints.  That means if you have changes in the lookups, you will need to add more inserts or updates to the DDL diff that DBIC makes for you.  Also, if you need to do updates, this introduces possible pain regarding you primary key fields if you are using some sort of auto increment or uuid system to create these, since you might not know in advance the values in these fields.

It's been recommended to me that you can use an 'enum' type in the database instead of table look ups.  That way you do indeed get to manage the data as part of the DDL, but you still end up having to use conditionals a lot for all the special rules.

At least with DBIC, you can mitigate this a bit by making classes and resultset constricted to a particular query (so you can make a Schema::Result(Set)::State::California that actually can hold the rules for California).  This still feels a bit ugly to me since you still have to do all the manual table inserts.

Anyway, that's my Friday rant.  Thoughts and suggests very welcomed.  For such a common issue a commonly shared solution we can put into code would be fantastic. 

As I was researching a followup to my previous post  and reading all your comments, as well as comments from reddit I can't help but notice what seems to be a distractingly large number of comments aimed at the very conservative Chinese outfit that I am wearing in my profile picture.  Some of you might know I lived in China for a few years and that my wife is Chinese.  So I choose to celebrate her culture in the traditional manner, despite the fact I am not particularly traditional myself.

Now, if you know me at all you'd know I care little for what I consider surface matters (Socrates is my favorite Philosopher).  I find it confusing that so many people seem to think it's important, and I'm actually insulted that some people seem to be comparing my picture to some controversial pictures used in a recent RoR conference (you can google for "Matt Aimonetti code like a porn star" but you might get stuff that's not work safe.)  I think there is a big difference based on the fact I could understand how some women, particularly in a male dominate industry like IT, could feel assaulted by the mentioned presentation, while I find it impossible to understand how someone could be rationally likewise disturbed by my picture.  I guess some people have very thin skins, or are highly distractable.

However it's seems to be sapping conversation about the topics I am writing about and draining time from people who jump up to defend me, when they could be using that time to write something useful.  I imagine some of them are probably tiring of the strange controversy as well.  I am here to talk about Perl, about Perl Programming, about the future of our shared ecosystem of practices, about where out community is going.  Not about cultural practices and intolerance. 

Thoughts on the matter appreciated. 

Recently with all the renewed effort on the part of Perl programmers to be better public advocates of our language, I've noticed an uptick in counter productive and quick frankly hateful comments directed at Perl.

I'd really like to find out why.  I'd also like to be reasonable and think there's actually something rational behind all this.  Several ideas come to mind.  For example, is all this negativity based on Perl's perception as spagetti code?  Is that just rumor or have many of you out there been required to step in and take over projects with difficult codebases?  Is it due to the fact that Perl 6 is still in development?  Has it to do with something many of you have experienced on the job?  Or maybe you've tried to get help from various Perl communities and gotten flamed?

I really have trouble understanding where this is coming from.  Personally, I use Perl because I find it works with my brain and not against it.  I've not found this to be the case with PHP, Python or Ruby.  I have nothing against any of those languages, but it makes little sense for me to force myself to relearn and rebuild my current skillset in a different language that doesn't really offer me a compelling reason to do so.  I'm deeply invested into the Perl community as a contributer to several important open source projects and that's what I enjoy about my job.

However the negativity is something I'd like to address if possible, because all this hate directed at Perl can affect me personally in terms of future employment and job options.  This directly affects my family, my wife, son and dog, so I take it very seriously. 

Because to be quite honest, the combined popularity of the most common scripting languages doesn't come close to countering the Java/C# juggernaut, and as someone committed to open source / free software this is an additional worry for me.  Neither of those languages are particularly committed to software freedom.  As far as I am concerned Sun (or Oracle now I guess) and Microsoft give some lip service to software freedom since that's a popular thing nowadays, but it's minimal and tactical in nature.  Free free to argue with me if you think I'm wrong, but I just don't see a deep and fundamental commitment.  Now, maybe you don't care about software freedom, but I'm old enough to remember the days when being a programmer nearly always involved proprietary tool chains and expensive certifications.  If it wasn't for the threat posed by developers and advocates of free software there would be no Java Community Process, no Mono, nor would there likely be quality free version of IDEs like NetBeans, Eclipse or the various free 'Express' versions from Microsoft.  There would be no free, online versions of documentation.  So it really seems to be we should be working together, not against each other. 

Several possible points of collaboration that suggest themselves would be standards for testing, deployment and management of applications.  Also, we often depend on a core set of external libraries, such as MemcacheD, xml and xslt parsing libraries and so forth.  It would be great if we worked together to improve these and to ensure that the language bindings were of consistent and high quality.

Each of our communities has different strengths and weaknesses.  The Rails group are great at advocacy and Ruby has a reputation for delivering nice, clean code.  Python really focuses well on education and ramping up newcomers.  Perl has great testing and deployment tools, as well as CPAN.  Why are sniping each other when working together would have so many benefits?

Introduction

This blog is a proposal and request for comments regarding adopting the XDG Filesystem Hierarchy as a option for managing all the non code data composing a Catalyst application.

The Problem

Right now when you create a new Catalyst application the non code data by default goes to either {home}/root (for templates and static stuff) or {home} (for configuration files), where {home} is the root directory of the application.  So you get a directory structure like:

MyApp
    myapp.conf
    /t
    /root
        /static
    /lib
        MyApp.pm
        /MyApp
            /Model
            /View
            /Controller

Now, this {home} directory is something of a hack, since we use Catalyst::Utils:home() to try to figure it out based on certain expectations.  Perl doesn't have this idea of {home} built into it.  If your application is 'installed' (via cpan or make install), we guess the location based on the physical address of the application modules (whatever you got that is inheriting from Catalyst).  If it's not installed (which is the common case when you are developing and just running the development server or tests) it walks the directory structure looking for a Makefile.PL or a Build.PL and then decides that's good enough to call {home}.

Since this method can be a bit flaky, a lot of people are recommending that you use File::ShareDir  (see here for a good overview).  This module intergrates well with Module::Install and leverages the fact that a Perl module can have a share directory associated with it.  Using this, you might create a directory structure like:

MyApp-Web
    /etc
        myapp-web.conf
    /t
    /share
        /static
    /lib
        MyApp.pm
        /MyApp
            Web.pm
            /Web
                /Model
                /View
                /Controller

I also modified the directory hierarchy a bit to reflect the growing consensus that your Catalyst application should ideally live one level further down from your application root.  In this case I choose 'MyApp/Web.pm' which seems to be the most popular choice and one that is semantically meaningful.  This represents the idea that your MVC layer should be the thinnest possible over your true domain and interface logic, which sits in the MyApp directory.  I also moved the configuration files to {home}/etc since that makes sense from people used to finding configuration in /etc

Although this is an improvement, it still suffers from several issues.  First of all one problem with File::ShareDir is that it can only find the share directory for installed applications.  For the common case where you are actively developing, or running tests, you still need some code like Catalyst::Utils::home() to guess the directory for you.  In this way it's not much better than what Catalyst::Utils::home() provides out of the box.

Also, when your share data is installed into the perl library path, this means that your application server (or user running apache mod_perl or fastcgi) would need the correct level of access to the path.  This complicates configuration.  This setup is this is not what most Unix administrators will expect.  There are reasonably well defined norms for where your configuration should go (/etc or ~/.config) as well as where the logs go and all that. 

Although you can override the {home} directory with environment variables, this is not ideal if our goal is to minimize installation hassle and make everything work well out of the box.  It complications your installation for users as well as configuration the web servers that will run the code.

It also complicate customization.  For example, let's say I am using the MojoMojo wiki and want to run three instance of it.  Each instance will have unique configuration and I want to slightly modify the theme files for each.  Right now, the only way I can do this is via the method of overridding the environment variable for home for each running instance.  Although this works, this is a 'roll my own' approach that is likely to vary from administrator to administrator, making it more difficult to onboard new admins due to the uniqueness of each application.  I strongly feel that we should have clear standards for all the most common case deployment issues, since this reduces errors, speeds deployment as well as counter the argument I often hear that Perl is hard to maintain.  A standard will also help grow a set of best practices surrounding deployment issues which we can document and promote.

Proposal

This is a case where Perl is not well leveraging existing norms, which really goes against the grain for us, considering CPAN with it's "reuse, recycle" mantra is one of our primary claims to fame.  My recommendation is that we adopt an existing standard and make this available as a plugin or set of roles for Catalyst.  The most relevent standard is the XDG Filesystem Hierarchy which exists specifically as a standard for where installed applications put configuration and data files, both locally that users can overide as well as global stuff that only admins should touch.

Although this standard is aimed at Linux, it's fairly straighforward and similar methods are employed by Windows Server and MacOSX Server so that is should be possible to create a pluggable support mechanism that is broadly applicable.

the XDG Filesystem Hierarchy defines some environment variables and defaults for the most common types of non code data, as well as offers a system for separating user configuration from global configuration.

I recommend you review the standard, since it's very short, but here's a summary.  The standard defines 4 enviroment variables useful to us:

XDG_DATA_HOME

These is the location of data oriented files that a user running the application should be able to customize (or will be customized during installation or use of the application).  By default these go into "~/.local/share".

XDG_CONFIG_HOME

Similar to XDG_DATA_HOME but specifically for configuration files.  Defaults to "~/.config".

XDG_DATA_DIRS

Takes a string of paths (delimited by ":") where to local for systemwide data.  These could be things like templates or static assets that shouldn't be changed by users and that would be shared by all instances of the application.  The default is: "/usr/local/share/:/usr/share/".

XDG_CONFIG_DIRS

Like XDG_DATE_DIRS but for configuration.  Defaults to "/etc/xdg".

The way I'd see this working is that if the application we being run in development mode, we'd first look for files local to the application file path, and then fall back to looking at the XDG defined directories.  Additionally, we'd probably need some boilerplate install scripts that authors can use to prompt for the desired path information (which rational defaults).  So our application distribution would possible look like:

MyApp-Web
    /t
    /etc
        myapp-web.conf
    /share
        /local
       
    /lib
        /MyApp
            Web.pm
            /Web
                /Model
                /View
                /Controller

And during installation we'd copy "MyApp-Web/share/local" to "$XDG_DATA_HOME/myapp-web" and "MyApp-Web/share/" to "$XDG_DATA_DIRS/myapp-web" (we'd either just copy to the first one in the path or prompt at install time).  Handling configuration would be a bit trickier.  My thougth here is that we'd copy "MyApp-Web/etc/*" to "$XDG_CONFIG_DIRS/myapp-web" but when running the application would like in both XDG_CONFIG_DIRS and XDG_CONFIG_HOME, merging both to allow locally overriding of the configuration.   

Overall I believe this will give us a smoother and more professional installation experience, make it easier to administer Catalyst applications and help start a best practices dialog.

Thoughts, criticism, abuse welcome :)

So in my attempt to add in some support for Dependent Type programming to Perl Moose programming I'm at the part where I need to integrate this into declaring your Moose based Perl objects.  To recap, with a Dependent Type Constraint you can declare a constraint like:
 

    subtype Range,
      as Dict[max=>Int, min=>Int],
      where {
        my $range = shift @_;
        return $range->{max} > $range->{min};
      };

    subtype RangedInt,
      as Dependent[Int, Range],
      where {
        my ($int,$range) = @_;
        return ($int >= $range->{min} && $int <= $range->{max});
      };

    RangedInt([max=>10, min=>1])->check(5);  ## Is fine
    RangedInt([max=>10, min=>1])->check(15);  ## Is NOT fine

You can then use this in a Perl class like so:

    class MyClass {
      has young_adult_age => ( is=>'ro', isa=>RangedInt[min=>18,max=>34] );
    }

And you can instantiate this like so:

    MyClass->new(young_adult_age=>20);

But the following will throw an exception:

    MyClass->new(young_adult_age=>40);

Also, if you are using MooseX::Declare you can use these in your method body signatures:

    method height(RangedInt[min=>10,max=>100] $height) { ## Do something here }

All this is supported in the current repository version you can review here.

So far so good.  However, it's been requested to be able to fulfill a dependency requirement via $self when using dependent types in a class.  My proposed syntax for this is to make the 'isa' and 'does' attribute options similar to default, where if the attribute is lazy and the value is a coderef, we de-reference it and pass $self.  So, for example, let's say you create some type constraints and a class that uses it like so:

    use Set::Scalar;
    use Email::Valid;
   
    subtype Email,
      as Str,
      where {
          return Email::Valid->address($_);
      };

    subtype Set,
      as Dependent[class_type 'Set::Scalar', Any],
      where {
          my ($set, $item) = @_;
          foreach my $element ($set->members) {
              return unless $item->check($element);
          } 1
      }
     
    subtype UniqueEmail,
      as Dependent[Email, Set],
      where {
        my ($email, $set) = @_;
        return !$set->has($email);
      }

    class Person {
        has email_set => (is=>'ro', isa=>Set[Email] );
        has email => (is=>'ro', isa=>UniqueEmail[Set[Email]] );
    }

Ideally, when you try to instantiate this class, the attribute 'email' would be bound to attribute 'email_set', in such a way that it's value should be unique in that set.  To achieve this, I am proposing an attribute trait that allows something like:

    class Person {
        has email_set => (is=>'ro', isa=>Set[Email] );
        has email => (
            is=>'ro',
            isa=>sub {
                my $self = shift @_;
                my $set = $self->email_set;
                return UniqueEmail[$set];
            },
            traits=>['Dependent'],
        );
    }

And at validate time, the coderef would be dereferenced with $self, similarly to the way the default option works.  We'd do the same for the 'does' attribute.  And to make it even more similar to default, I'd support coderefs here without lazy, which would not pass $self, but would execute at runtime.

For myself, I'd rather have another solution, but maybe I just don't fully grasp the use cases.  Comments on the above proposals very welcome. However, if you are wanting something very different, you should give me enough specification for me to get it. 

==UPDATES==

It's been suggested that instead of overloading 'does' and 'isa', to use a new options, such as 'generate_isa' or similar. Should be more sane and play nicer with other attributes that expect isa and does to behave a certain way.

One thing which has come up from time to time on the Moose Type Constraint wish list is the ability to more easily define type parameters.  For example, it would be great if we could very simple create an Integer type that is bound by a range.  For example:

    subtype Range,
      as Dict[max=>Int, min=>Int],
      where {
        my $range = shift @_;
        return $range->{max} > $range->{min};
      };

    subtype RangedInt,
      as Dependent[Int, Range],
      where {
        my ($int,$range) = @_;
        return ($int >= $range->{min} && $int <= $range->{max});
      };

    class MyClass {
      has young_adult_age => ( is=>'ro', isa=>RangedInt[min=>18,max=>34] );
    }

And you can instantiate this like so:

    MyClass->new(young_adult_age=>20);

But the following will throw an exception:

    MyClass->new(young_adult_age=>40);

This gives your Moose based Perl programming a bit more flexibility!

Right now this is not on CPAN, but you can see it, play with it (and help me shed light on any of the corner cases) at: https://jules.scsys.co.uk/gitweb/gitweb.cgi?p=gitmo/MooseX-Dependent.git;a=tree

At this point the code is shaping up, although there's a bit of ugly stuff in the type constraint code.  Even coercions work as you might hope.

Pending responses to the above syntax, remaining things before CPAN is primarily some attribute traits so that we can expose $self to a dependent type in a class.  This would allow something like:

    class Person {
      has people => (is=>'ro', isa=>PeopleResultSet, required=>1);
      has id => (is=>'ro', is=>UniqueID[PeopleResultSet], traits=>['Dependent']);
    }

The idea being that the attribute 'id' would be bound to the attribute 'people' so that you could not create a new Person that had a pre-existing ID.  The attribute trait stuff is  still pending and feedback, test cases / use cases and of course code are very welcome.

UPDATED:  My apologies for rebranding the date to anyone who has a working aggregator, but seems that whatever the settings for the Ironman aggregator, it wasn't picking up my post. Maybe I'm supposed to say, "Perl Programming now..."