Dan's Drivelings » ColdFusion

Automatically encrypting and decrypting data with ColdFusion ORM and annotations

Dan Skaggs — Mon, 04 Apr 2011 15:19:59 +0000

I’m writing an application that uses ColdFusion’s ORM features heavily. Various fields in my database deal with Personally Identifiable Information (PII) and need to be encrypted to meet regulatory requirements. I’ve been mulling over the best way to deal with keeping the data encrypted while in the database but have it be readily usable when loaded into an entity. None of the scenarios that came to mind felt right or could be implemented without an extensive amount of “work-around” code and I just wasn’t willing to go down those paths.

I put the question out to the folks that follow me on Twitter and got a couple of responses, one of which was from Mark Mandel who suggested using annotations. Now, I’ve heard the word annotation mentioned, but had never had the time/opportunity to research what they were or how they were used. That was a few weeks ago and in the meantime, I’d gotten busy focusing on other things and just got around to thinking about the encryption thing again a few days ago. What I found out astonished me.

In ColdFusion CFCs, an annotation is nothing more than an additional attribute you put on a component, property or function tag (and maybe others too, but these will do for my scenario). To test this theory, I added the attribute encrypted=”true” to the password property of my User entity (the user accounts for people that are authorized to log into and use the application) and issued an ormReload() command to recreate the ORM entities. That worked without an error, but at this point it really didn’t do anything except just add a bit of metadata to that property tag.

The secret to the automatic encryption and decryption comes when you combine the encrypted=”true” annotation with the event handling methods provided by ColdFusion ORM. ORM provides the ability to run your own custom routines at various points throughout the ORM request cycle–preInsert(), preUpdate() and postLoad() are three of the ones that I took advantage of. In essence, I used the preInsert() and preUpdate() methods to automatically replace the clear-text value of properties that had the encrypted=”true” flag with an encrypted version of that value. Similarly, I used the postLoad() method to reverse the process and replace the encrypted string that is stored in the database with the clear-text value.

I’ve included the code bits that I used below.

The password property of the User entity:

<cfproperty name="password" 
     encrypted="true" 
     type="string" 
     length="100" 
     required="true" 
     notnull="true" />

The ORM event handler methods:

<cffunction name="preInsert" output="false" access="public" returntype="void">
     <cfset encryptProperties() />
cffunction>
 
<cffunction name="preUpdate" output="false" access="public" returntype="void">
     <cfset encryptProperties() />
cffunction>
 
<cffunction name="postLoad" output="false" access="public" returntype="void">
     <cfset decryptProperties() />
cffunction>

I then created the encryptProperties() and decryptProperties() methods to loop through the properties of the given object looking for any properties that had the encrypted=”true” attribute set and perform the associated transformation when/if one is found:

<cffunction name="encryptProperties" 
     output="false" 
     access="public" 
     returntype="void" >
 
     <cfset var props = getMetaData( this ).properties />
 
     <cfloop array="#props#" index="prop">
          <cfif structKeyExists(  prop, "encrypted" ) AND prop.encrypted >
               <cfset variables[ prop.name ] = 
                 encrypt( variables[ prop.name ], "", "AES" ) />
          cfif>
     cfloop>
 
cffunction>
 
<cffunction name="decryptProperties" 
     output="false" 
     access="public" 
     returntype="void">
 
     <cfset var props = getMetaData( this ).properties />
 
     <cfloop array="#props#" index="prop">
          <cfif structKeyExists(  prop, "encrypted" ) AND prop.encrypted >
               <cfset variables[ prop.name ] = 
                 decrypt( variables[ prop.name ], "", "AES" ) />
          cfif>
     cfloop>
 
cffunction>

To make the above approach even more useful, I put all those methods in a CFC named AbstractEntity.cfc that all my ORM entity CFCs extend. So, now whenever I need to add another column to store encrypted data (no matter which entity I need them in as long as that entity extends the AbstractEntity.cfc), I define all the attributes for the property as normal and simply add the encrypted=”true” attribute to the property and ColdFusion handles the encryption and decryption for me automatically. The end result is that not only does this happen transparently to me as the developer, the clear-text values that I need to interact with in my application are encrypted when stored in the database as well as while in transit to/from the database.

Adding ModelGlue’s event API to ColdFusion Builder’s code insight

Dan Skaggs — Mon, 07 Feb 2011 16:54:23 +0000

For the last year or so, all of my ColdFusion development projects have been built using the Model-Glue MVC framework. Over time, you get to where you know the most commonly used methods that are used to interact with the framework by heart. Sometimes there are lesser-used methods that you have to go look up. All the time (for me at least) I’m trying to find a way to write code faster and with less errors.

ColdFusion Builder has done a very nice job of providing code insight for ColdFusion tags, functions and CFC methods. This is especially true if you have a server configured in the “Servers” panel and mapped to your CF Builder project as it then will provide code insight for your own CFCs that you create on the page. However, when using Model-Glue, the “event” object is created for you and is always there. Because it’s not explicitly created on the page, CF Builder can’t provide code insight when you need to interact with it. However, it only takes a couple of settings in your project to make CF Builder aware of the event object and start providing help for it.

Here’s the process:

Right click on your project and choosing “Properties”.
In the left pane of the window that comes up, click on “ColdFusion Variable Mappings”.
On the right side, click the “New” button and enter the following values into the boxes
- Variable Name: event
- Mapped To: html.ModelGlue.gesture.eventrequest.EventContext
Press the “New” button and enter the following values into the boxes
- Variable Name: arguments.event
- Mapped To: html.ModelGlue.gesture.eventrequest.EventContext
Press the Apply button then the OK button
* Note that the value in the “Mapped To” box is the actual dot-notated path to the EventContext.cfc file from your CF Builder project root. I happen to have my webroot files in a folder named “html” under the project root (see screenshot #1 below).

Once you have those settings saved, any time you type “event.” or “arguments.event.” you’ll get the list of methods contained in the Event object. Of course, this doesn’t only work with Model-Glue. Any CFC that you regularly use the same name with can be configured this same way.

I’ve attached some screenshots for reference. If you have any questions or something isn’t working, feel free to ping me.

Converting one company to CFML using an OSS CFML engine

Dan Skaggs — Fri, 03 Dec 2010 19:26:06 +0000

There have been some lively discussions on Twitter today centered around the adoption of open source CFML engines in various types of businesses. I can’t speak to any kind of trends anywhere, but I wanted to share my experience with a project I was involved in last year that used Railo as the CMFL engine.

My brother, Andrew, is the head of the IT department for a firm that sells books and other supplies to school libraries all over the country. When he took over the position, he inherited a web site built on Java using the Struts framework. The original site had been in place for quite some time and the ownership decided it was time to refresh the site. The main issue (besides the outdated design) with the site was that it was so difficult to update. I’m not a Java developer by any stretch of the imagination, but from what I understand, even the simplest update required the site to be recompiled. The actual implementation of the site was fairly brittle as well, so new features were generally not even considered for fear of breaking something that was currently working.

Andrew had previously done some ColdFusion work with me on some of my other projects. He approached his management about contracting with me to help them replace their java web site with one built on CFML. During the initial requirements gathering and planning phases, the decision was made to use Railo as the CFML engine for the site. The site was launched a few months after Railo released their first open source version which appealed to them as they have several systems within the company that are open source. Cost was not a major concern (Railo being free vs the cost of a standard license of Adobe ColdFusion) but it was a consideration as this project did have a fixed budget. When we started the project, Adobe ColdFusion 8 was the most current release. Budgetary constraints probably a few more factors I wasn’t privy to led to the decision to use Railo as the CFML engine for the site.

Our experience developing with Railo was very positive. We built the site using Mach-II, Transfer, ColdSpring using a PostgreSQL database backend. We had one little issue with Mach-II when we plugged in the Mach-II Dashboard module into our main config. Peter Farrell and I traced it down and found that it was due to a difference in how Railo parsed XML vs how Adobe ColdFusion did. I worked with the folks at Railo and they got the problem sorted out in one of the next releases.

The site went live the first of August, 2009. Between January 1 and August 1 of 2010, the site experienced a 562% increase in revenue generated when compared to the same time period of 2009 (the YTD time frame leading up to the new site going live). They’ve also seen a dramatic increase in the amount of traffic to the site since the new version went live (although I don’t have exact percentages at my fingertips to share). Just as importantly, in the months sice the new site went live, they have been able to add dozens of new features to the site–both publicly-available features as well as features for administrators and sales people that just would have been too difficult to try to weave into the previous java struts based application.

Finally, while this “success story” has Railo as one of its key components, the bigger issue in my mind is the fact that a formerly java-centric shop has converted to using CFML successfully to run their business and provide a better experience for their customers and employees.

Disclaimer: I’m not on either side of the Railo/OpenBD/Adobe CF argument. I’ve not had a client since that project that’s given me the option to use Railo so all of my work since then has been in Adobe CF but I certainly wouldn’t dismiss Railo straight away if there were an option to use it on a future project. The intent behind this blog post is simply to provide one real-world example of Railo being an entry point into CFML development, not to take either side of the Twitter discussions that have been going on today.

**Update: Corrected the incorrect timeline of when Adobe ColdFusion 8 came out and removed some inaccurate statements about speed of Railo vs the Adobe CF version out at the time (which was mistakenly stated to be MX7). Thank you Adam for pointing out the error.

Dynamically set ColdFusion ORM datasource

Dan Skaggs — Mon, 20 Sep 2010 16:46:59 +0000

I’m a relative novice using ColdFusion’s ORM features having done just one “real” project so far that took advantage of it. I’m working on an application that needs to be able to set the datasource for each request based on the URL that the customer is using to access the site. For example, if a customer visits http://customera.demoapp.com I need the application to use the DSN named “dsn-customera”, http://customerb.demoapp.com gets “dsn-customerb” and so on.

Normally this wouldn’t be an issue–you’d do some sort of logic in the onRequestStart() method in Application.cfc and set a request or session scoped variable to the name of the DSN and go from there. However, when using ColdFusion’s ORM functions, this approach doesn’t work since “this.datasource” is configure in the pseudo-constructor at the top of Application.cfc like so:

1 2	<cfset this.name = "myApp" /> <cfset this.datasource = "dsn-myapp" />

The first way that I thought of solving this was to set this.datasource=”demo” in the pseudo-constructor and then overwrite the value in the onRequestStart() method like so:

<cfset this.name = "myApp" />
<cfset this.datasource = "dsn-myapp" />
 
<cffunction name="onRequestStart" output="false" >
 
    <cfset local.calculatedDSN = "" />   
 
    
 
   <cfset this.datasource = local.calculatedDSN />
 
cffunction>

This did in fact override the value in this.datasource as evidenced by dumping the “this” scope at the start and end of the onRequestStart() method. However, CF’s ORM system always used the value set in the pseudo-constructor even though the this.datasource value was changed successfully. After scratching my head for a while, I posted a message to the cf-orm-dev Google group with what I was trying to do and the results I was seeing.

Turns out, I was on the right track, I just had my logic in the wrong place. After exchanging a few messages with Brian Kotek, Sumit Verma, and Andrew Scott, I learned that CF’s ORM system is configured before any of the event handlers specified in Application.cfc are fired.

In the end, the answer to what I was needing to do was to put a call to a custom, private function in the pseudo-constructor to set up the customer-specific DSNs and then CF’s ORM system would be configured dynamically on each each request for the correct customer database. The only other gotcha is that you have to have a unique application name set up for each customer in order for things to be cached correctly. Attempting to set this.datasource per request while using a single application name resulted in all requests using the datasource specified by the logic on the first request to the application.

The code ended up looking something similar to this:

<cfcomponent output="false">
 
    
    <cfset setupCustomerValues() />
 
    
 
    <cffunction name="setupCustomerValues" output="false" access="private" 
       returntype="void" hint="">
 
       <cfset local.customerName = "" />
       <cfset local.customerDSN = "" />
 
        
 
        <cfset this.name = "myapp-#local.customerName#" />
        <cfset this.datasource = local.customerDSN />
        
    cffunction>

This is obviously demo/proof-of-concept/blog-brevity type code so there are a couple of things to note:

This method creates a separate ColdFusion application in memory for every client. If you have a large number of objects that are created and cached at application startup and a large number of customers hitting the same server, you could/will run into memory issues. In my case, this was less of a problem than having to deploy a completely separate code base for each customer using the application.
The demo code doesn’t take into account any kind of caching of the results of the customer-specific logic. Obviously in a production application, you wouldn’t want to execute that logic on every request, but cache the result after the first execution for that customer (in my example, the first request to a particular subdomain)

Returning ColdFusion 9 ORM objects with JSON

Dan Skaggs — Mon, 19 Jul 2010 16:26:26 +0000

Last week I was working on a client’s project that makes heavy use of ColdFusion 9′s ORM features. Everything we’d done with ORM up to that point had been going really well and I continue to be impressed by the amount of time ORM saves me in development. I had gotten to a point in the project where I needed to be able to use AJAX calls from jQuery to manage some of the data in the database. Based on my experiences using a RemoteFacade.cfc to feed data to a Flex app, I thought it should be pretty easy. Several hours later, I realized that my particular use case for this was anything but easy.

What I eventually found was, when trying to load an object using an AJAX call to RemoteFacade.cfc, any object that had a property that was an array of other objects was not returned to Javascript. It wasn’t even that it was sent across as an empty property. The property absolutely didn’t exist in the JSON returned by the CFC. I used the ColdFusion debugger to verify that the array existing in the object immediately prior to the statement and then examined the JSON return using Charles Proxy only to find out that those properties weren’t in the JSON. All the normal string, numeric, date, etc type properties were returned just fine, but any properties composed of other objects were missing.

As near as I can tell, the culprit is the serializeJSON() method that ColdFusion uses to convert complex objects into JSON notation. It doesn’t seem to work correctly on ORM objects. To test my theory, I manually built a structure and inserted each simple property from the ORM object. I then created a key in the structure that was an array and manually populated several items in the array each with a structure of data. When I returned that from the CFC as JSON, the array and all the structures it contained came across the wire just fine.

I’m not sure if this is a bug in the serializeJSON() method or if serializeJSON() wasn’t meant to handle objects. Whatever the case, this issue leads to quite a bit of extra work if you need to return an object via JSON and include its properties that are composed of other objects. I’d definitely be interested in hearing from anyone who has successfully done this with ORM and AJAX to see if I’ve done something incorrectly in my code.

Building Advanced Workflows with ColdSpring at cf.Obective()

Dan Skaggs — Sat, 24 Apr 2010 22:53:03 +0000

Thanks to all the folks that came to my Advanced Workflows with ColdSpring session today at cf.Objective(). The conference this year was outstanding and I’m honored to have been chosen to speak. If you’re interested in getting the slides and code, I’ve uploaded a zip file that includes everything I used in the session.

Embracing “New” in ColdFusion 9

Dan Skaggs — Fri, 16 Apr 2010 17:54:08 +0000

I’m not sure how I missed this before but among the long list of things added to ColdFusion 9 is the ability to create a CFC using the “New” syntax. Up until now, to create a new object from a CFC we’d use the createObject() method like so

<cfset Team = createObject("component", "model.Team") />

Now, using the New keyword, we can shorten that to

<cfset Team = New model.Team() />

But wait, there’s more!
When you use the New keyword, ColdFusion will automatically look for and run any init() method that exists in the CFC. It also respects any arguments that your init() method specifies, meaning that if you have a Team.init() method that can accept team name, color and manager arguments, you can build them right into your the same line of code used to create your object like so

<cfset Team = New model.Team( "Bumblebees", "Yellow", "Benny Bee" ) />

Likewise, as you’d expect, named arguments are still supported. So if you had to send in arguments out of order or needed to omit some optional arguments at object creation, you could do

<cfset Team = New model.Team(
              manager="Benny Bee",
              color="Yellow",
              name="Bumblebees") />

I personally am going to make a point to start using this in my code because I can never seem to spell the word “component” correctly in the createObject() method.

My ColdFusion Anniversary

Dan Skaggs — Thu, 18 Mar 2010 12:41:33 +0000

Well, not quite yet, but since I’m going to be out of pocket the next several days, I wanted to put this out now. Monday, March 22 will mark 10 years since I began my “journey” with ColdFusion. That day in 2000 was my first day in a class called “Fast Track to ColdFusion”.

The class was taught by Sean Hedenskog in San Jose, CA where the headquarters for the company I was working for at the time was located. That company had decided on Allaire Spectra 1.0 as the basis for an enterprise content management system so several of us went through the Fast Track to ColdFusion course in preparation for Spectra training some weeks later.

I know there are are a lot of people out there who have more time with ColdFusion than I do. I started with version 4.5 and, if memory serves, our company was one of the first to deploy ColdFusion and Spectra onto Solaris-based servers. Even so, I feel like I’m somewhat of an old-timer these days. That’s a good thing however, since it means we’re getting more and more new people using ColdFusion.

A lot has changed in the 10 years since I started that class. Coming from a strictly HTML background with a little bit of classic ASP thrown in, even the beginner class posed quite a challenge for me. Simple things we don’t give second thoughts to such as calling a custom tag (there were no UDFs or CFCs then remember) were hard for me to get my head around. I remember having an especially difficult time trying to figure out how arrays and structures worked

Along the way I’ve been privileged to associate with some very talented and helpful folks. Not long after we started the Spectra project, my company brought in some folks from the Allaire consulting group. Matt Stevanus was one of those folks and quickly became a great friend and mentor. We kept in touch periodically after that project was over and a little over 3 years ago I got the opportunity to work with Matt again when I started consulting with Universal Mind.

A couple years ago I met another guy who would help shape my professional career–Dan Wilson. Dan is widely known these days as a driving force behind the ModelGlue project. Dan and I had an opportunity to work on a long-term project together and he and I’ve enjoyed working with him on various things ever since. It was Dan that really got me over the “OO hump”, convinced me that I needed to start blogging more and to submit topics to various conferences to speak on.

Finally, there are dozens of people out in the ColdFusion community doing blog posts, open source projects, MeetUps and more that have contributed to my success. To everyone, I say a hearty “THANKS!” (and my mortgage company loves you too).

So, here’s looking back at a great 10 years while looking forward to many more ahead!

Be careful with the ‘local’ scope when migrating from CF8 to CF9

Dan Skaggs — Mon, 08 Feb 2010 14:24:15 +0000

One of the really nice “fixes” included in ColdFusion 9 from a developer’s perspective is the inclusion of an implicit “local” variable scope into which variables created within the body of a tag are placed by default. Previously, developers had to manually add a “var” keyword to variables that should only exist within the confines of the function.

One of the ways of simplifying this that gained some traction among various developers prior to the release of ColdFusion 9 was to “var” a single variable at the top of the function as an empty structure then store any additional variables needed in the function inside it. Many folks, myself included, named this structure “local” so that it would be readily apparent that the values inside were local to that function. This approach worked fine and dandy on ColdFusion 8 and below.

I recently migrated one of my clients to ColdFusion 9 and not long after the client started getting isolated reports from his people having javascript errors in a data management application that uses AJAX-driven forms talking to CFCs. At first these were very isolated and we weren’t able to reproduce the error, but, as time went on, the reports became more widespread. As I was troubleshooting this over the weekend, I discovered that we were getting javascript errors when trying to interact with this RemoteFacade CFC about 40% of the time.

Using Firebug, I was able to watch the results come back from ColdFusion and noticed a very odd trend. Approximately 60% of the time, the JSON returned from the remote CFC call was as expected. In the other 40%, one of the main data structure names was an arbitrary, machine-generated name instead of the name we had specified in the code.

Here is an example of what we were expecting in the JSON returned from the RemoteFacade.cfc method:

{"DATA":{"CANEDIT":true,"RECORDDATA":{"Field1":"value1"} } }

This is an example of what we would get back from the same request when we would see the javascript errors

{"___IMPLICITARRYSTRUCTVAR5":{"CANEDIT":true,"RECORDDATA":{"Field1":"value1"} } }

See the difference? Our “DATA” key was named completely differently which caused javascript to throw some error saying that variableName.DATA did not exist.

After looking over the ColdFusion code for quite a while and doing some step debugging with FusionDebug I had an idea. I changed all the function-specific structures I had been using from:

<cfset var local = structNew() />
   ....
<cfreturn local />

to:

<cfset var ret = structNew() />
   ....
<cfreturn ret />

Once I changed those throughout the code base and reinitialized my application, all the javascript errors that we’d been experiencing across multiple request types went completely away.

I have an unconfirmed theory that in using the name “local” for my structure, ColdFusion was sometimes getting “confused” on what to return–ColdFusion’s built-in local scope or the method-specific variable I had named local. I don’t really have any way to prove that beyond a shadow of a doubt, but when I made the change, all my errors went away, so I decided chalk it up as a lesson learned for future development and move on to the next problem. Needless to say the client was happy that the issue was fixed and I can say I learned something that day.

ColdFusion 9 caching settings to watch out for

Dan Skaggs — Sun, 31 Jan 2010 15:14:09 +0000

Like a lot of developers, I’ve got this pet project I’m working on in whatever spare time I can find between client engagements, home maintenance, family obligations, etc. I’m using it as an opportunity to work with some of the new features of ColdFusion 9 (ORM mainly), ColdFusion Builder Beta and features in development for the next release of ModelGlue 3.

Recently, Bob Silverburg has been working on a significant overhaul of the scaffolding feature used by ModelGlue to automatically create CRUD forms for the various data objects in your application. Particularly exciting to me is that you can now override the built-in code templates with your own. Bob wrote a proof-of-concept application that uses the excellent cfUniform custom tag library to build standardized forms and validations (see my previous post on cfUniform if you’re not familiar with it). Since I’m pretty particular about how my project files are arranged, I proceeded to place the css, javascript and image assets into the folders where I wanted them and use ColdSpring to create a configuration bean to pass to cfUniform when I called it. That’s where the trouble began.

I had to make a couple of changes to the code generated in the custom scaffold CFC in order to have cfUniform see the custom configuration that I had set up. No matter what I did, when the scaffolding engine generated the code for the view and the XML fragment for the event-handler, the changes I made inside the CFC weren’t included. I spent a couple hours scratching my head, tracing the request cycle, restarting my local ColdFusion instance and always got the exact same code that was in Bob’s original example CFC. Finally, I decided to change the name of the CFC and update the associated bean configuration in ColdSpring. On the next refresh, I saw my changes reflected in the code generated by the scaffold!

With that in mind, I checked the settings on the Caching page of that instance’s CF Administrator. Sure enough, the Cache Template In Request, Component cache, and Save class files options were checked. I cleared those check boxes, pressed the Clear Template Cache Now and Clear Component Cache Now buttons below and have had no trouble since. Obviously there are situations where you want these enabled, but rarely ever should they be needed on a local system being used for development.

So, the moral of my painful story–if you’re making changes to code that’s not being reflected when you test browse your application, don’t forget to check the settings on the Caching page in CF Admin. It just might save you a couple hours and a few gray hairs.