AWS CodeBuild - HowTo

Amazon Web Services' CodeBuild is a managed service that allows developers to build projects from source.

Typically CodeBuild is used as part of your CI/CD pipeline, perhaps along with other AWS tools like CodeCommit, CodePipeline and CodeDeploy.

This blog will explore the use of CodeBuild to build the Bedrock project and update a yum repository.  Along the way I'll detail some of the things I've learned and the path I took to automating the Bedrock build.

Using Apache's Swiss Army Knife - mod_rewrite for RESTful endpoints (part III)

This is the last blog in my series on Apache's mod_rewrite.  In my previous blogs I've explained a little bit about how you can use this Apache module to redirect URLs both internally and externally.

Using Apache's Swiss Army Knife - Part I

Using Apache's Swiss Army Knife - Part II

I've provided a few examples and talked about some of the gotchas that I've encountered.  Today we're going to discuss how to use mod_rewrite to provide an API endpoint that is an alias for your web server's name without creating another virtual host.  In most cases, it makes sense to create another virtual host configuration file to implement your application API, however in the event you don't have access to Apache's configuration files, you can use mod_rewrite rules in an .htaccess file to achieve nearly the same operations.

Serving up POD & The five Cs of Good Documentation

I'm a fanatic about documentation.  As a developer that has been involved in creating tools for other developers all my career, I understand the importance of good documentation.  Creating good documentation is not easy, which is why so few people write good documentation.  Here's an easy to remember grading system for your documentation.  Does it meet the five Cs test?

 Is your documentation...

1. Correct
2. Complete
3. Comprehensible
4. Current
5. Convenient

I probably don't need to go into too much detail here, but if ain't complete, correct, comprehensible and current it may be worse than no documentation at all.  If it ain't convenient it really doesn't matter how correct, complete, comprehensible or current it is, now does it?  It's not easy meeting all 5 Cs all of the time.  Most likely your documentation will transition through various permutations of those Cs during the software development lifecycle.  So how can we at least make our documentation more convenient and accessible and at least allows us to make an attempt to meet the 5Cs?

Using Apache's Swiss Army Knife - mod_rewrite (part II)

This blog was supposed to introduce some tricks to allow you to create a new subdomain for your RESTful endpoints, but I think before we launch into more details about creating your RESTful API endpoints I should share a few relevant discoveries regarding using mod_rewrite that might be of interest.

Using Apache's Swiss Army Knife - mod_rewrite for RESTful endpoints

As I've been exploring using Bedrock for creating RESTful APIs, one of the requirements for a well crafted API is to create appropriate URIs that make sense.  I've read the phrase "you don't write RESTful APIs you DESIGN them" in more than one blog.  This usually boils down to exposing the API in a logical and hierarchical manner.  Apache's mod_rewrite is a versatile and useful tool for allowing you to create meaningful, well crafted endpoints that also work with Bedrock.

<include --optional>

Bedrock 2.4.1-10 now includes a new option of the <include> tag.  The new --optional argument, will prevent Bedrock from throwing an exception if your include file is not found.  This might be useful for those situations where you might want to conditionally include a file.

<include:foo --optional>

There a couple of old school ways of achieving the same thing.

<if -e --file ($config.INCLUDE_DIR + '/foo.inc')>
  <include:foo>
</if>

or

<try>
  <include:foo>
<catch>
</try>

Either way you slice it, the old school way seems forced.  The --optional tag is admittedly syntactic sugar, but a welcome sweetener!

Cookieless Sessions - Automatically Login Your Users

The default Bedrock persistence store, implemented via the $session object has the ability to create a token (session identifier) you can use to automatically log your users into your applications.

This is a useful feature for resetting passwords and perhaps creating convenience links for fast access to application specific pages that would normally require a user to login to the application.  Here's how it works...

Bedrock + Barewords = Bug

This morning, after being sufficiently caffeinated with a full pot of coffee, I was able to isolate a Bedrock bug that has been bugging me.  Bedrock was failing to throw an exception on what I would consider a syntax error (missing right end tag >).  

  <null:foo $baz.some_method()

  <var $foo>

Bedrock however, happily parsed the expressions and went along it's merry way...argghhh!

Is it a bug in Bedrock?  Probably, but IMHO it's really a problem with Bedrock's loosely defined grammar.

RESTful APIs with Bedrock (part V)

This is the 5th in a series of blogs on using Bedrock to create a RESTful API.  So far we've checked off several requirements for building a RESTful API.  We've been able to demonstrate the basics of creating a RESTful API using Apache and Bedrock.  We've satisfied requirements #1 through #4 and #6.

1. Parse a URI
2. Determine the operation to be performed
3. Determine the result format requested by the client
4. Retrieve, validate, and de-serialize client data
5. Retrieve data from the back end
6. Serialize data to send back to client
7. Set HTTP response headers
8. Handle errors

We'll round out the discussion by showing how Bedrock can satisfy the 3 remaining requirements.

RESTful APIs with Bedrock (part IV)

To recap our 8 requirements of a web development framework for creating our RESTful API:

1. Parse a URI
2. Determine the operation to be performed
3. Determine the result format requested by the client
4. Retrieve, validate, and de-serialize client data
5. Retrieve data from the back end
6. Serialize data to send back to client
7. Set HTTP response headers
8. Handle errors

In yesterday's blog we talked about using Apache directives to help us create meaningful URIs and using Bedrock to interpret the extra path information and HTTP method in order to satisfy requirements 1 & 2.

Today we'll examine #3 and #4 and touch on #6.

Perl Regular Expression Capture Groups

After cursing Bedrock for not giving me access to capture groups in regular expressions, I finally bit the bullet and figured out how to implement this in a somewhat sane manner.

RESTful APIs with Bedrock (Part III)

The other day I blogged about using Bedrock to create a RESTful API.  Today I'm going to talk about setting up Apache in order to support a RESTful interface based on URIs.  Last time we pulled these requirements for a REST API framework from Michael McClennen's talk:

1. Parse a URI
2. Determine the operation to be performed
3. Determine the result format requested by the client
4. Retrieve, validate, and deserialize client data
5. Retrieve data from the back end
6. Serialize data to send back to client
7. Set HTTP response headers
8. Handle errors

Let's configure Apache to help us with #1, and #2 as we implement a hypothetical RESTful interface for a contact database.

First, we'll create an Apache RewriteRule that maps our desired API's base URI to a Bedrock page.  We add the directives below to our Virtual host configuration file.

RewriteEngine On
RewriteRule  ^contacts(/.*)?$ rest-api/contacts-api.jroc/$1

Using this rule URIs that begin with contacts will now be rewritten to point to a URI that implements our REST interface in Bedrock.  The Bedrock file contacts-api.jroc is a Bedrock script (page) that returns a document of type application/json.  By convention (and as implemented by the Bedrock handler), any Bedrock script with an extension of .jroc will be considered to return a JSON document.

That's just another way of saying that Bedrock will set the mime-type header to application/json for you automatically.  Of course you could have named the file contacts-api.roc and then set the Content-Type header yourself.

<null $header.set('Content-Type', 'application/json')>

Files in the rest-api directory are configured using Apache directives so they will be processed by the either the Bedrock mod_perl Apache handler or the Bedrock handler implemented as a CGI.

<Directory /var/www/vhosts/myhost/htdocs/rest-api>
# Bedrock - mod-perl for .roc (if mod_perl)
  <IfModule mod_perl.c>
    AddHandler    perl-script .roc .jroc .html
    PerlHandler   Apache::Bedrock
  </IfModule>

  Action bedrock-cgi /cgi-bin/bedrock.cgi virtual

  <IfModule !mod_perl.c>
    AddHandler    bedrock-cgi .roc .jroc
  </IfModule>
...
</Directory>

We can now use the PATH_INFO environment variable to interpret our URI since all the stuff after  contacts/ in our URI will be considered extra path information and presented to us in that environment variable...well sort of.   As it turns out, when Bedrock is configured using the CGI version, PATH_INFO will contain the filename (contacts-api.jroc) as well as the extra path info.  For this reason, Bedrock provides an environment variable named BEDROCK_PATH_INFO only available when the Bedrock handler is configured as a CGI script that contains only the extra path information.

PATH_INFO => (/env.roc/foo)
BEDROCK_PATH_INFO => (/foo)

Our REST API script can now grab the extra path information necessary for us to interpret the URI.

<null:path_info --default=$env.PATH_INFO $env.BEDROCK_PATH_INFO>

Under mod_perl we'll use the PATH_INFO variable, but as a CGI handler we'll use the value Bedrock provides as BEDROCK_PATH_INFO.  Since the BEDROCK_PATH_INFO variable is only set when the Bedrock handler is a CGI, our $path_info will be set  using the default value under mod_perl.

The URI gives us the context, but what about the method?  Using Bedrock's $env object that exposes  environment variables to your Bedrock pages, we can examine the HTTP method sent by the client.  By testing  $env.REQUEST_METHOD we can determine the HTTP method.  The value will probably be either GET, PUT, POST, or DELETE, as set by the client and defined by your API.  It would then be easy enough to test that value along with the URI to determine the operation to be performed.  That is to say, the operation to be performed will be a function of the HTTP method and the URI.

<if $env.REQUEST_METHOD --eq 'GET'>

# /contacts/friends
   <if $path_info --eq '/friends'>
   ...
   <elseif $path_info --eq '/business'>
   ...
   <else>
     <raise '400'>
   </if>

<elseif $env.REQUEST_METHOD --eq 'PUT'>
...
</if>

This now satisfies our first two requirements we discussed last time, namely:

• Parse a URI
• Determine the operation to be performed

Next time: Determining the result format and de-serializing our input data.

RESTful APIs with Bedrock (Part II)

Reading the Perl blogs I saw a link to a Michael McClennen RESTful talk.  It was an interesting introduction to those with no background on REST and offered some insights and interpretations based on his experience.  The talk does a good job balancing an informational talk on RESTful interfaces in Perl with a gentle sales pitch for his new CPAN module (Web::DataServices).

First off, anyone that publishes on CPAN has my great respect.   Clearly this author knows his stuff, shares it and that's a very good thing.   I'm not sure I will become a user of this module, primarily because of the dependencies (Moo, etc), but more to the point I'm not (yet) convinced one needs a complete RESTful framework that sits on top of a web development framework.

Having said that, I reserve the right to change my mind and I imagine the author spent a lot of time thinking of that as well and ultimately concluded this his effort was worthy of the problem.   Congrats to Michael on publishing his module and adding another building block or at least reference for Perl developers.  I admire that effort and wish I had the time and energy to do the same.

So, do we really need a RESTful API framework?

I'm not sure.  Why don't I think we need a RESTful framework?  Well, I would hope that that the web framework itself that was chosen is robust enough to implement the RESTful architecture and your API without a lot of overhead or hair pulling.  After all, the RESTful architecture is supposed to reflect the stateless and simple nature of the web.  Sort of, I guess.  Paraphrasing a slide from Michael's presentation, there's not too much one needs from a framework to produce a RESTful API.  Here's what we'd like from our framework in order to implement our API.

A RESTful API Using Bedrock?

As I've written before, writing AJAX services in Bedrock is actually pretty trivial.  In fact, Bedrock's support of the .jroc extension which automatically sets the mime-type of the output to application/json makes creating an AJAX widget to return the current environment for example, look something like this:

<var --json $env>

Not sure how it could get any easier than that...and with a properly configured Bedrock installation, AJAX services that access information stored in MySQL tables is equally as easy.

<sqlselect "select * from customers" --define-var"customers"></sqlselect>
<var --json $customers>

I've recently started looking at using Bedrock to prototype a RESTful interface and I've been very encouraged about the simplicity here as well.  Several features of Bedrock make creating a RESTful API pretty easy.

• Ability to interpret the environment variables (URIs), etc.
• JSON
• Access to POST data
• Ability to directly set status
• Ability to interpret and set HTTP headers

I've been studying up on creating robust, secure and compliant RESTful interfaces and think I can use Bedrock to prototype a RESTful API.  Some interesting resources I've found so far include:

REST Video Tutorial
REST API Design Rulebook
API Security Considerations

So what might a Bedrock based RESTful API look like anyway?  Consider this API:

/accounts      => returns a collection of accounts
/accounts/{id} => returns a single account's record
/books         => returns a collection of books
/books/{id}    => returns a single book record

Let's give this a shot in Bedrock...

<sink><null:uri $env.BEDROCK_URI>
<null:parts $uri.split('/')>

<hash:out>

<if $parts.[1] --eq 'accounts'>
 <if $parts.[2]>
   <sqlselect "select * from account where id = ?"
              --define-var="out"
              --bind=$parts.[2]></sqlselect>
 <else>
   <sqlselect "select * from account" --define-var="out></sqlselect>
 </if>

<elseif $parts.[1] --eq 'books'>
  <if $parts.[2]>
    <sqlselect "select * from book where id = ?"
               --define-var="out"
               --bind=$parts.[2]></sqlselect>
  <else
   <sqlselect "select * from book" --define-var="out></sqlselect>
  </if>

</if></sink><var --json $out>

Okay, this is a really simple and featureless example that doesn't consider authentication, or setting status on error, but you get the idea.  I'll be blogging about this subject as soon as I do the deep dive, but it looks promising.

Redis & Bedrock

Been playing a bit with Redis, a really cool in-memory database based on keys and values.  It's a key/value database on steroids and has a lot of applicability for web applications that need to scale or just want better performance.

A Perl module is available on CPAN, appropriately named Redis.  One might therefore be tempted to do something like:

<plugin:Redis server localhost:6379>
<null $Redis.set('key', 'value')>

Great idea, however it turns out that didn't work.  Is Bedrock broken? Hmmm...let's find out.

Perl modules like Redis that utilize a large number of AUTOLOAD subroutines and do not implement an appropriate can method are not going to present a code reference when Bedrock's parser calls the module's can method...so as a recognition of this possibility the Func.pm module within Bedrock's parsing does this:

my $funcref;

$funcref = $obj->can($func);

...

if ( $funcref ) {
    @ref_v = $funcref->( $obj, @funcargs );
  }
  else {
    &log_message(
     undef,
     "$self: eval function in case implemented via AUTOLOAD\n\@ref_v = $obj->$func(\@args);"
    );

    eval '@ref_v = $obj->' . $func . '(@args);';

    if ( $@ ) {
      if ( $@ =~/TagX::Scalar/ ) {
  die "attempt to invoke possibly non-existent method ($func) on a possibly non-existent scalar\n";
      }
      else {
        die "$@\n";
      }
    }
  }

Hey, umm...what's that? @args? Shouldn't that be @funcargs? Yup....bug.  Fixed in 2.4.1-snapshot today.  Okay, so anyway...

What can we do with Bedrock and Redis?

Well, generally we can use it as a persistent storage mechanism similar to the way we use the $session object.

<null $Redis.set('foo', 'bar')>

...on one page and then...

<var $Redis.get('foo')>

...on the next...

Of course that example is like saying you can use a brick to prop open a door!  There are lot more interesting and exciting things you can do with Redis.

How about counting page hits from unique IPs?

<null $Redis.pfadd('home-page', $env.REMOTE_ADDR)>

Using Redis' HyperLogLog (requires Redis 2.8.9) facility we can add a member and let Redis keep track of counting unique hits.

<var $Redis.pfcount('home-page')>

Or resetting a counter at midnight by specifying a time value (seconds since the epoch) to the expireat function.

<sqlselect "select unix_timestamp(date_add(curdate(), interval 1 day)) exp">

<null $Redis.expireat('home-page', $exp)></sqlselect>

...and of course we can store serialized data to Redis.

<null $Redis.set('env', $env.json()>

and deserialize it later to a Bedrock object...

<null:user_env --json $Redis.get('env')>

Yup...this is cool!  ;-)

Pretty Print JSON objects

Added --pretty as an option to the <var> tag for pretty printing JSON objects.  Changes pushed to git master.

<sink><null $header.content_type('application/json')>
<hash:foo red 1 blue 2 green 3>
</sink><var --pretty --json $foo>

...would yield

{

   "green" : "3",
   "blue" : "2",
   "red" : "1"
}

AWS with Bedrock

I'm starting to tinker with using Amazon's web services from Bedrock pages.  Embedding AWS in Bedrock applications is not new - particularly if you are running Bedrock on an EC2 instance, RDS (hosted MySQL) or you are using Amazon's S3 object storage system.   There are other Amazon web services that are particularly useful in any web development environment, exposing these to Bedrock pages looks interesting.

Plugin Bug

Fixed a bug in the <plugin>  tag.   Class names were not being parsed correctly if there was more that 1 level to the namespace.

For example, Amazon::SNS::VerifySignature was being incorrectly parsed as Amazon::SNS which happens to be valid Perl module.

Fix pushed to master.

Amazon SNS - Verifying Signatures

While creating a Bedrock example of using Amazon's Simple Notification Service, I was surprised to find that Googling for a Perl example of verifying the signature from an SNS notification message came up empty.  So I wrote my own...

Custom Error Pages

The default page when Bedrock encounters errors during the parsing of a page is error.roc and is distributed with the other core Bedrock files.  The page shows the developer the error and the contents of the page with each line numbered for easy correlation to the error reported by Bedrock.

Bedrock Error Page

You can customize the error.roc page if you'd like or replace it.  You probably don't want to spill code to users in a production environment.

To specify your own custom error page, set the BEDROCK_ERROR_LOCATION variable in the tagx.xml file to the fully qualified path of your own Bedrock file.

<scalar name="BEDROCK_ERROR_LOCATION">/var/www/html/error.roc</scalar>

Bedrock exposes the $ERROR object that has several methods you can use to report the error.

$ERROR.mesg() - the error message

$ERROR.line() - the line number where the error was encountered.

$ERROR.file() - the name of the file where the error occurred.

$ERROR.view_source() - an HTML representation of the source code.

Simple Example

This example uses the $ERROR object to display the error message, location of the error, and the file that was being processed when the error occurred.

<html>

<body>
<h1>Whoops! We're sorry, our slip is showing!</h1>
<pre>
message: <var $ERROR.mesg()>
location: <var $ERROR.line()>
file: <var $ERROR.file()>
source: <var $ERROR.view_source()>
</pre>
</body>
</html>

E-Mail Me the Error

Here's an example error page that will give the user a message and send an email to the developers notifying them of an error...I'm using the $SMTP plugin distributed with Bedrock to email the error message.

<html>

<body>

<h1>Whoops! We're sorry, our slip is showing...</h1>

<i>Thanks for your patience...our staff is on it!</i>

<plugin:SMTP localhost rlauer6@comcast.net rlauer6@comcast.net>

<null $SMTP.subject('A Bedrock Error has occurred!')>

<null $SMTP.header('Content-Type', 'text/html')>

<sink $SMTP>

An error has occurred <b><var $ERROR.file()>[<var $ERROR.line()>]</b>

<hr>

<b>REMOTE_ADDR:</b> <var $env.REMOTE_ADDR>

<b>SERVER_ADDR:</b> <var $env.SERVER_ADDR>

<hr>

<pre>

File: <var $ERROR.file()>

Line: <var $ERROR.line()>

<span style="color:red;"><var $ERROR.mesg()></span>

</pre>

</sink>

</body>

</html>

...and voila!  I get an email when an error occurs.