Western Skies

Why is my Rails app calling Solr so often?

2011-10-14T21:44:00.000-07:00

I work on the back-end of a Rails app that uses Solr via Sunspot. Looking at the solr logs, I could see the same item being added/indexed repeatedly sometimes right before it was deleted from solr. I didn't write the code, but I was tasked with figuring it out.

Glancing at the main path of the code didn't show anything obvious. I figured the superfluous solr calls were happening via callbacks somewhere in the graph of objects related to my object in solr, but which one(s). Again, I didn't write the code, I just had to make it perform.

I hit on the idea of monkey-patching (for good, not evil) the Sunspot module. Fortunately, most/all of the methods on the Sunspot module just forward the call onto the session object. So, it's really easy to replace the original call with anything you want and still call the real Sunspot code, if that's what you want to do.

This is so easy to do that I even did it the first time in the rails console. In that case, I was happy to abort the index operation when it first happened. So, I whipped this up in a text file and pasted it into the console:

module Sunspot
class <<self
    def index(*objects)
      raise "not gonna do it!"
    end
end
end

Then, I invoked the destroy operation that was triggering the solr adds, got the stack trace, and could clearly see which dependent object was causing the index operation.

For another case, I needed to run a complex workflow in a script to trigger the offending solr operations. In that case, I wanted something automatically installed when the script started up, and I wanted something that didn't abort - all I wanted was a stack trace. So, I installed the monkey-patch in config/initializers/sunspot.rb and had a more involved index function:

    def index(*objects)
      puts "Indexing the following objects:"
      objects.each { |o| puts "#{o.class} - #{o.id}" }
      puts "From: =============="
      raise rescue puts $!.backtrace
      puts ===========\n"
      session.index(*objects)
    end

That last line is the body of the real version of the index method - like I said, trivial to re-implement; no alias chaining required.

Maybe there's some cooler way to figure this out, but this worked for me.

enjoy,
Charles.

Rails/Rspec does not clean up model instances on MySQL

2011-08-18T09:50:00.000-07:00

I recently solved a thorn in my side relating to some Rspec tests in our code base when running on my development machine using MySQL. For some reason, some instances that were created using Factory Girl weren't getting cleaned up, which in turn would cause subsequent test runs to fail because of duplicate data. So, I'd DELETE the whole tables from the MySQL prompt. I looked in the test.log file, and I could see the save points being issued before the objects were created, but they weren't getting removed at the end of the test.

I didn't have a lot of time to look into it, and I didn't know where to look - Rspec, Factory Girl, Rails? So, in the short-term, I just added after_each calls to destroy the objects. And, I moved on.

Then, I was dumping schemas in MySQL using SHOW CREATE TABLE in order to analyze some tables and indexes, and I noticed the storage ENGINE flag on the tables. I went back and looked at the tables in my test database that were giving me trouble, and, of course(?), they were MyISAM rather than InnoDB. So, transaction rollback (used to clean up after tests) didn't work.

I changed the storage engine on those tables (ALTER TABLE t1 ENGINE = InnoDB), commented out the manual clean-up code, and voila! It works right now. Pretty obvious in retrospect, but I didn't even know where to start looking in our stack.

I hope this helps some other poor souls, too.

Charles.

Freeing up phone space on Android

2011-04-26T10:02:00.000-07:00

For the last couple of months my Motorola Droid running Android 2.2.2 has been complaining about being "low on space" for the phone, not the SD card. I pruned some apps, but that didn't help much. Things really came to a head this morning when my phone was so low on memory that it was no longer downloading email.

I found this article to be quite helpful -
http://www.androidcentral.com/monthly-maintenance-keeping-things-speedy

For me, the two big ones were Messaging and the Browser cache. I had a couple of threads in Messaging containing a number of pictures. Once I saved the pictures off to the SD card, I purged the threads, that freed up ~20MB. Clearing the browser cache freed another ~20MB, but that will probably evaporate again as the browser caches things.

Here's a minor whine about Android: the SD card and phone storage settings page tells you how big your SD card is and how much space is remaining, but the phone storage just says how much is left. Without knowing how much I had to start with, it's hard to know if, say, 20MB is a lot or not. As near as I can tell, Android seems to complain when the space is less than 25MB.

Update: I ran out of space again, and clearing the browser cache didn't help. After bumping around some more, first I discovered that in "Manage Applications" the one and only menu option is to sort by size. Doing that revealed that the new pig was the (post pay-wall version) New York Times application. It was using over 60MB of data space in the Phone Storage area. The app doesn't have a "clear cache" function, so I used the "Clear Data" button from within Manage Applications, and I was back in action.

enjoy,
Charles.

A Fix for "Exceeded MAX_FAILED_UNIQUE_FETCHES" in Hadoop

2010-11-05T17:19:00.000-07:00

In a project I'm currently working on, we're moving a bunch of our back-end processing to Hadoop. We started a two-node cluster: one master, one slave. That seemed to work fine. Then, we went to four nodes, and about the same time I was testing out a new Hadoop job. The (single) reducer was hanging with this somewhat cryptic message:

Shuffle Error: Exceeded MAX_FAILED_UNIQUE_FETCHES; bailing-out

I went out to the slave node and looked through the job logs, and I could see that it was timing out trying to transfer data from one of the other slave nodes - an upstream mapper node. Upon closer scrutiny of the the log file I realized that Hadoop was trying to transfer from the other slave's public IP address, which is behind a firewall that blocks public access.

Key take-away number one: when you're just starting out with Hadoop, if you only have one slave, you've only demonstrated one real communication path: master-to-slave. Your cluster isn't doing any slave-to-slave transfers because everything was on the one slave. Also, our initial job had no reducer, so it ran fine on the new, 4-node cluster because it was still only master-slave communication.

For some reason, the mapper slave was advertising the location of the map output data via its public IP address. My first attempt at fixing this problem involved the dfs.datanode.dns.interface configuration parameter (and it's mapred equivalent). This tells Hadoop that when a process (mapred or dfs) wants to figure out it's host name, use the IP address associated with the given interface. (You could even have dfs and mapred using separate interfaces for additional throughput.)

This failed for me because I had one interface with two addresses, not two interfaces. I dug through the Hadoop DNS code (org.apache.hadoop.net.DNS - God, I love open-source: you can just look for yourself) and saw that if there is one interface, the code loops through the IP addresses and performs reverse DNS lookups and takes the first successful result. I was fortunate in that the private IP address was coming up first in that enumeration of the IPs on the interface, but it still wasn't working. I talked to our system admin/configuration guru. It turns out that our hosting provider doesn't provide reverse DNS for those private IP addresses. We could have set up our own DNS server for just these reverse lookups, but there was a brute-force option available to us.

You can bypass all of Hadoop's efforts to automatically figure out the slave's host name by specifying the slave.host.name parameter in the configuration files. If that is set, Hadoop will just take your word for it and use the name you provide. Now, in theory, this might be onerous - it means you have a different configuration file per-slave. However, our cluster is configured and maintained via Puppet. So, our puppet master just tweaked his Puppet script, and we never looked back.

Take-away number two: Exceeded MAX_FAILED_UNIQUE_FETCHES could mean a simple connectivity problem. I'm sure there are other possible causes, but an inability to connect between slaves is comparatively simple to troubleshoot.

enjoy,
Charles.

Django vs. Grails

2010-04-06T14:56:00.000-07:00

When I came up with the Five Technologies in Five Weeks project, I hadn't intended to compare any of the technologies to each other directly. My original ordering for the technologies didn't have any head-to-head match-ups, but I just finished my Grails project, which was "right after" (modulo interruptions) my Django/GAE project. And, while I was working on Grails, I kept comparing it to Django, and I felt compelled to write up my observations.

"All things being equal, which they never are" (Manager Tools), I'd choose Django over Grails for a new, green-field project. However, given the constraint to run in a Java environment, I'd gladly choose Grails over the other Java/J2EE frameworks I'm familiar with. (Django on Jython would be a contender as would Wicket, but I don't have hands-on experience with either.) This is not a slam on Grails.

Here are some of the reasons I prefer Django over Grails:

Development Speed (not runtime performance): running (a trivial number of) unit tests for Grails on my machine took about 15 seconds (with Folding@Home in the background). Django unit tests typically only take me a couple of seconds (for a trivial number of tests). Integration tests in Grails were 30-40 seconds; Django might be 5 seconds. Although Grails will reload the running webapp when you make a change, sometimes things are really buggered, which requires a full restart. That seemed to happen more often in Grails than Django, and restarts take ~20 seconds vs. 2 seconds.
Compared to the 5-7 minutes I've seen WebSphere to restart an application, Grails is blazingly fast. But the 15-30 seconds that many operations take in Grails is long enough for the mind to wander, which is not good.
Update: a commenter pointed out the (under-documented) interactive mode of Grails. That speeds the edit-test-edit loop considerably.
Voodoo: both Grails and Django do some voodoo behind the scenes to reduce the amount of chimp programming (e.g., non-DRY boilerplate) the programmer has to do. This voodoo involves topics of the high priests like meta object programming that mere mortals typically don't have to worry about. Maybe it's just that I don't have as much "time behind the wheel" with Grails, but I feel like its voodoo leaks out a bit and is too voodoo-rific. A Grails programmer has to be aware of which methods are dynamically added to a class in order to separate unit tests (no voodoo) from integration tests (full voodoo). And, although I like the power of dynamic finder methods like User.findByName(), it kinda bugs me that the code for that doesn't exist somewhere that I can see (you know, on punch cards!) You don't see the dynamic methods on a Django class, but there are a lot fewer of them, so it seems less voodoo-rific. Again, maybe more time behind the wheel of Grails would make me feel more comfortable.
(As an aside, when I taught Python and Django in Spring 2009, the students didn't even notice the voodoo of fields on models in Django until I pointed it out to them. Then, I exploded their heads with MOP.)
View Technology: I have a long-standing personal preference for templates instead of ASP/JSP-like mark-up languages, and I'd lump Grails GSP pages in the latter category. Back in the day, Jason Hunter wrote an essay called The Problems with JSP that really stuck with me. GSPs are much, much better than the Model 1 JSP pages that Hunter talks about, but they still feel similar enough to make me think about using something like Velocity with Grails. Django kicks templates up a notch by having inheritance with templates, which I really love. (Even if Django didn't invent template inheritance.)
Dynamic Language Issues: This is a real unfair comparison because I've been using Python for 15 years, but I felt that errors in Grails/Groovy were more cryptic and hard to find compared to Python. If I typo the named parameter to a method in Python, it lets me know immediately. More than once in Grails I misspelled a named parameter to a constructor, which failed silently and then lead to a validation failure later when I went to save the object. Some of that is from lack of experience with Groovy/Grails - i.e., learning what error messages really mean. But still, Python seems to fail in a more helpful way. (I'd say Python fails gracefully, but if anything it's the opposite: very loud - "you dumb-ass, there is no parameter called recipent" when I misspelled recipient.)

I don't mean for this to be a hit-piece against Grails; I really like Grails. I look forward to using it some more. It's just that, for what it's worth, I like Django more.

5Tech - Week 2: Grails

2010-04-06T13:54:00.000-07:00

For the second project and technology in five weeks I chose Grails. I was very curious to try Grails on on a "real"/non-tutorial project to confirm its usability and productivity. On my last contract gig, I was in a non-development role (configuration management) on a project that was using JSF 1.1/Spring/Hibernate and WebSphere, and their development looked really painful by my standards. For example, they'd spend 5-7 minutes to redeploy the app just to inspect some HTML change in the JSF page. I was looking to Grails to provide a much more productive environment.

The project involved creating a system to process events and route them to users. It was inspired by a code base that I worked on for a previous client. (This was done with their permission.) They have a large code base that includes "application functionality" like this event routing, as well as a lot of "technical functionality" that they rolled themselves years ago before modern tools like Spring and Hibernate were created and became mainstream. In a way, this was a small prototype to research the feasibility of reimplementing the application functionality on a more modern platform - Grails.

Analysis
All in all the project went very smoothly. Because Grails is comparatively mature, there exists a fair amount of documentation, including numerous books. I leaned heavily on Grails in Action by Smith and Ledbrook. I was able to follow their examples and adapt them to my application easily. There just weren't any serious gaps or surprises.

Using Grails to create domain classes (database entities) was a breeze. It's so nice to just declare the fields and their constraints and have Grails "do the right thing." You don't have to bother with annotations, let alone XML configuration files. Creating controllers to process web requests is trivial, and being able to scaffold them to get the basic CRUD functionality in place immediately was very conducive to high productivity.

I also liked the idea that services are a first-class concept in Grails (along with controllers and domain classes), which makes it very easy to program with them. Grails wires them into the classes that use the services via Spring, but again, you don't have to monkey with Spring's applicationContext.xml file. Finally, adding the REST interface for incoming events was almost trivial, especially since I already had the service in place.

Grails (like Rails) treats testing as a core concept, not something you wave your hands at after the fact. It has the concept of unit and integration tests, and there are a number of functional test plug-ins, too. I did a fair amount of unit and integration testing, which was a real live-saver. Due to the very dynamic nature of Groovy, many of my typos were not caught in the compile phase, but exercising the code in tests did catch those. I had very few issues when I ran the actual web application.

A lot of developers are unclear on the difference between unit testing and integration testing. For better or worse, Gails makes some clear distinctions. A lot of the Grails voodoo (e.g., dependency injection and adding dynamic methods to classes) is not available in unit tests, or you have to add it yourself via mocking or manual injection. Thus, there is a simple distinction: if it runs under "test-app -unit", it's a unit test, otherwise it's an integration test. This bit me once early on when I wrote a test that needed that higher-order functionality, but I put it in a unit test. First I got a null pointer exception because the service hadn't been injected, then I got a missing method exception because the save method on the domain object hadn't been added dynamically. However, the fix was simple enough - move the test to the integration folder, and it ran fine with -integration. Then, I copied it back, added a few mocks (very easy in Grails), and I had a unit test, too.

In terms of "project management," this step of the Five Technologies suffered from some "life happens" distractions. Rather than running Monday through Friday, it ended up being Tuesday through Monday, and some of those days were a bit short-changed. When I was on task, the Pomodoro Technique continued to be effective, and I replaced my dead watch/timer with a dedicated Android application called Pomodoro Tasks - it's even open-source.

In conclusion, Grails was another success for the Five Technologies in Five Weeks project. I got the application functionality done that I expected. Grails was very usable and productive - no nasty surprises. What's next? Probably, NetBeans Platform because I have documentation for it already whereas I'm waiting on some Android books.

5Tech - Week 1: Google App Engine

2010-03-24T11:15:00.000-07:00

For the first of the five technologies in five weeks, I picked something easy - Google App Engine using Python and Django. As someone who's been using Python for 15 years, there was no language learning curve. And using the Django helper for app engine package allowed me to leverage my Django experience. So with a minimal learning curve, the results were basically all good.

The project involved creating a simple application to monitor web sites to check if they are up or down and notify the user about status changes. As such, the core of the application isn't even a web app. In fact, I've implemented the same thing a couple of different times as a standalone program in Python or Java. However, to run a standalone application like that requires a server where it can run, and that's not something I've always had access to. GAE's cron service provides the ability to run checks periodically - just like the main loop in my standalone applications, and GAE provides a large number of notification options, although I only used email initially. The application does have a web interface for configuring checks and viewing the status

Project Analysis

My biggest problem developing the application was the fact that I was working with three separate but overlapping tools/frameworks: GAE, Django, and the Django helper. These all worked fine, but when I wanted to do some task, I didn't know where to look for the "right" way to do it. After wasting a bunch of time searching for things only to find that the Django way was the right way, I just quit asking and adopted the "try Django first" strategy. And then, the first time I went to apply that I got burned - there are some slight differences in how the filter method is handled in Django helper versus "native" Django. But for the most part, Django first is the way to go. In addition to the online documentation for these three tools/frameworks , I used Programming Google App Engine by Sanderson, which covers all of them - I highly recommend it.

Although my plan with Five Technologies is to use best practices, I am embarrassed to admit that I didn't practice much test driven development on this project despite having "home field advantage" with the technology - Django. Some of that is due to the exploratory nature of the programming - just figuring out which end is up, and some of it was the confusion over how to do tests - as noted above, the answer is "the Django way". Also, I found a bug in the app engine helper when loading a fixture, which hung me up - I'll be submitting a patch shortly.

One (non-technical) practice that I adopted was the Pomodoro Technique, and that worked pretty well. The 25 minute mini-sprints were really nice to contain technology-induced ADD. However, the watch I was using as my timer died during a pomodoro which lead to a very long and productive pomodoro.

In terms of the bigger goals of Five Technologies, I have another confession: I did not restrict myself to one week. I created version 0.1 and deployed it to the cloud within one week, but the week following this project I went to the Java Posse Roundup, and I kept working on the Django app, even though I should have been focusing on Java. It's just that after a week of working on GAE, I had built up some good momentum and didn't want to quit. I fear that this will be a real problem for the next projects because I won't have the luxury of continuing to work on whatever technology when I begin the next project.

In conclusion, the first technology experiment was a success, even if I wasn't dogmatic. What's next? Most likely Grails. Stay tuned.

Update: I forgot to mention something cool I learned about - schema migration, or the lack thereof. Before I began the project I was fretting about schema migration on GAE because I've been too lazy to learn something like South, and therefore I do schema migrations at a SQL command prompt. Obviously, there is no SQL problem for a NoSQL database like Google's BigTable. Then I forgot about the issue, but half way through I looked up and said, "hey, I haven't been doing any migrations, but this all works." Duh! - like many NoSQL databases, BigTable is schema-less, so there is no schema to migrate. Problem solved. OK, the application has to be prepared to do with an attribute on a record/row that isn't present, but that code is basically the same as dealing with a NULL value that you might specify as the default value when you issue an ALTER TABLE to add a column. Also, you can still imagine scenarios where you might still have to do some sort of schema/data conversion, but without even consciously thinking about it, I managed to avoid those. That was cool.

Five Technologies in Five Weeks

2010-03-08T16:35:00.000-08:00

I am currently between consulting jobs, and during the down time, I have embarked on a project to learn five new (to me) technologies in five weeks. The reasons for doing this include:

Learn new things - this project is a variant on the "learn one new language a year" meme that's been going around. I'm just taking on five things (not necessarily languages) in much less than a year.
Bust some code - it's been a while since I've been able to do any hard-core coding. This will be a sprint which can blow out some cobwebs.
Improve my development practices by trying some new techniques and focusing on refining existing ones.

The technologies I plan on tackling are (subject to change):

Google App Engine (with Python, not Java)
Grails
Android
NetBeans Platform (not just using the IDE)
Griffon

I chose these technologies because I know of them but haven't actually built anything with them. Also, they are technologies that I'm interested in testing out to see how usable/effective they are and if I should pursue them further.

In a way, each week's technology is a bit like a "spike" in an agile project, only I'm not looking to evaluate/sketch out solutions to application problems but rather evaluate technologies in a more abstract sense. Although, in some cases (e.g., NetBeans Platform and Griffon) I have some application ideas I'd like to implement, and I really am spiking possible solutions.

For each technology, I've got a modest application in mind. I have (or will have) a series of story cards describing various aspects of the application I'd like to create. And then, I will sprint for a week to implement as many of the stories as possible. I'll also post at least one blog entry as a retrospective for each sprint.

I've been planning this for some time - ever since the end was in sight on my last contract. The most significant threat to this undertaking is not failing at one or more technologies, but rather if I find another contract before I've completed the technologies. (There are worse things than finding a paying job when you're currently between jobs.) Another known disruption to the "five weeks" is that I'm taking one week off to go to the Java Posse Roundup, which will technically make this five technologies in six weeks, but that isn't as snappy as five in five.

Hiring is Only for Managers?

2010-03-04T10:12:00.000-08:00

A friend of mine told me that he just met the new guy on their team. I thought it was odd that he was just meeting a new team member, so I asked if he wasn't around during the interviews, and he told me that the developers on the team never interview candidates - only managers do that. As near as I can tell, they do this to minimize the time required during interviewing. This is just wrong.

In the words of Manager Tools, "hiring is the most important job that a manager does" because, in part, failures pull down the whole team for months or years. And, team-fit is a crucial part of that interviewing process. All things being equal (which they never are), it's better to hire a technical 8 who has a 10 personality than it is to hire a technical 10 who was only an 8 (or less) personality. If nothing else, you can teach technical stuff, but you can't teach personality.

Dave Ramsey describes his lengthy, multi-round interview process that even includes dinner with spouses to ensure team (in the largest sense) fit. As lengthy as the process is, he points out that fixing a hiring mistake costs much more than the added time of proper interviews.

I'd even argue that this manager-only interviewing process produces shortcoming in the technical area, too, because the manager works off of a superficial checklist that s/he has to get through quickly in the interview. Thus, if a candidate is asked "do you know web framework X," and the answer is "no, but I know frameworks A through F, and I wrote a framework I call G," that candidate is treated the same as someone straight out of school who doesn't know any frameworks. This narrow-mindedness leads to hires that know framework X, but they store passwords as plain-text because no one told them not to, and none of them knows how to use MD5 to store a password (another story from my friend). These are what Erik Sink calls programmers not developers - you want (well-rounded) developers.

So, with apologies to Georges Clemenceau, interviewing is too important to be left exclusively to the managers.

JIRA Workflow Transition Displaying Wrong Text

2010-02-11T10:11:00.000-08:00

I've been doing some JIRA administration and customization for a client, and I ran into a problem that was driving me crazy. I created a new workflow to deal with an Issue Type called Change Request by copying an exsiting workflow. I renamed one of the transitions from "Close Issue" to "Decline CR," and I associated a screen with the transition (the reason for creating a new workflow).

The name of the transition ("Decline CR") displayed correctly in the administration page, but in the regular display page of a Change Request issue, the name of the workflow action was still the old value ("Close Issue") from the original workflow. I tried a few things to fix it, but nothing seemed to help. I even tried changing the workflow scheme and changing it back. Still nothing.

This morning while fishing around in the administration pages, I found the problem. The transition included an internationalization property, jira.i18n.title, that specifies what appears to be the title of the workflow action, closeissue.title. I picked up that property when I cloned the workflow - the ultimate origin of the workflow was the JIRA's default workflow, which was internationalized. Removing the property got the workflow action name ("Decline CR") to display correctly. (I'm not working in a multi-language environment, so I don't need any localization.)

World's Ugliest URLs

2009-10-01T20:58:00.000-07:00

I live outside the city of Monmouth, Oregon (beneath the Western Skies, of course). Our city's official website has URLs like:
http://www.ci.monmouth.or.us/index.asp?Type=B_BASIC&SEC={F6D36CB4-8AB1-4E2F-9F16-EEB14A3A83DD} - Things to See & Do
http://www.ci.monmouth.or.us/index.asp?Type=B_BASIC&SEC={6010A930-F666-42AF-A359-971AC53933A1} - City Government

OK, maybe they're not the ugliest in the world, but those bad boys have gotta rank way up there. Anyway, when I first heard Jacob Kaplan-Moss talk about pretty URLs in Django, I thought some of criticisms were a little nit-picky. Initially, I didn't see anything wrong with index.php, but I could see the basic point even before seeing Monmouth's URLs. When I converted the Evergreen Terrace Farms site from PHP to Django, cleaning up the URLs was something I was pleased with. For example, www.et-farms.com/animals/detail.php?name=bart became www.etfarms.com/animals/bart. So, I guess I have drunk the Kool-Aide, and I have become a bit of a URL snob.

Regardless of how picky you are about URLs, I think any sane person would agree that those URLs for the Monmouth site are just crazy. Just imagine trying to read one of those to your mom over the phone - "no, it's 42AF, and the A and the F are capitalized."

In my opinion, the saddest thing is that the site is a commercial product created by a company that boasts about how many governments they've sold it to. It would be one thing if some students created something like that for a senior project, but when you're charging people money for something like that you should at least not expose the ugliest of the ugly Microsoft crap from the depths of the implementation (I assume those are UUIDs generated by .Net). I guess I'm also disappointed that no one in the city even noticed those URLs before putting out taxpayer money.

Charles.

Presenting an Updated Version of OSS Jobs Talk

2009-09-16T20:49:00.001-07:00

I'm giving an updated version of "How to Get a Software Job w/o Experience" at Willamette University tomorrow, 9/24/09. It's a fun talk that I enjoy doing. I updated the SlideShare version, rather than uploading a new version, so the old version is publicly unavailable.

How to Get a Software Job w/o Experience

View more documents from Charles Anderson.

Mercurial for the Homeless

2009-08-23T21:33:00.001-07:00

Generally, I'm not one to talk about crazy dreams. You know those ones right before you wake up that seem like a Salvador Dali painting? Anyway, I had one the other day where either I was homeless living out of a car, or I was observing someone like that. Anyway, it dawned on me in the dream that Mercurial (or any DVCS) would be perfect for that situation because I figured that the homeless person would only have intermittent network connectivity, but he still needed to get work done.
OK, maybe the part about someone living out of their car "needing" to get coding done is a bit crazy, but I already admitted that it was a weird dream. I suppose jet-setting open-source gurus might feel kinda homeless from time to time, and they do need to make use of their time on planes, so maybe it wasn't so crazy after all. And, when you get right down to it, the two use cases are more similar than different, and Mercurial and other DVCSs excel at it.
For the record, the rest of the dream was related to work I'm doing for a client converting from CVS to SVN without necessarily having shell access to the server machines. I thought of a zillion ways where I could do something trivially in Mercurial but SVN (without shell access) just kept kicking my butt. I just hope shell access comes through, so the dreams can cease.

Charles.

Problem with Hudson as a Windows Service

2009-08-06T14:10:00.000-07:00

Setting up Hudson to run as a windows service is wicked easy. However, at a client we ran into a problem with the user that the service runs as. By default, the service runs as the Local Service (or maybe Local System) user. Also, by default Hudson runs out of the .hudson directory under the user's home directory (in Documents and Settings).

This ran fine until someone logged into the console. Hint number one that there was a problem was that Windows took forever to log the user in while there was a huge amount of disk activity. Then, suddenly Hudson reverted to its state from several weeks before - recently added users and builds were missing. Looking around I could see that many users had a .hudson directory (as well as Maven .m2 directories), which made no sense - only the Local Service user was running Hudson.

As near as I could tell, logging in on the console was somehow changing the value of the Local Service user's home directory. And for some reason, Windows thought everyone should have a .hudson directory. (Thus the long time to login - Windows was copying the Hudson directory to the new user.)

Anyway, the solution was to create a real directory for Hudson - C:\Hudson, and to point Hudson at that by setting the HUDSON_HOME environment variable. To make this as easy as possible, I just set it via the Control Panel as a system-wide environment variable. I filled the directory with the contents of the .hudson directory in my home, as it seemed to have most/all of the recent users and jobs.

Also, while fixing all of this up, I created a limited (non-administrator) user to run the service. This is just good security that was skipped when Hudson was initially set-up (by someone else) as a quick proof-of-concept prototype.

enjoy,
Charles.

Book: The Principles of Successful Freelancing

2009-07-14T20:21:00.000-07:00

The Principles of Successful Freelancing by Miles Burke
ISBN 978-0-98004552-4-6

The Principles of Successful Freelancing is a comprehensive introduction (if that's not a contradiction of terms) to striking out on your own as a freelancer. This book is perfect for someone who is considering moving to freelancing or possibly for someone just starting out.

Mr. Burke covers all of the basic areas of starting and running a freelance business. He discusses how to set up your business and your office, how to sell your services, how to manage your money, and how to give good customer service, which is ultimately the most important aspect of a personal freelancing business. He also addresses how to balance work and life beyond work, which is hard in general and specifically hard in a one-man shop. He concludes with something I haven't seen in a "start you own business" book - where to go next. Do you want to remain as a one-man shop, do you want to grow into a "real" business, or do you just want to "retreat" to the old 9-to-5 job? I don't recall a book like this consider the option of going back to the grind.

Each chapter concludes with two "case studies" - Emily and Jacob. These two characters represent two very different people who might want to go into freelancing. The studies at the end of each chapter explain how these personality types might react to the issues and challenges discussed in the chapter. This device helps the reader envision how he or she might deal with the issues discussed.

Early on, I got the mistaken impression that this book was a bit fluffy. The typography has a fair amount of white space, and it looks kinda arty rather than serious and dense. (OK, I grew up with punched cards and line printers. When's Matlock on?) But, by the time I finished the book, as I looked back across it, I really couldn't think of anything that wasn't covered. Sure, there are whole MBAs built around marketing, and this book only has one chapter on it, but the Mr. Burke provides a perfectly reasonable introduction to the subject. I think I got this "fluffy" mis-impression because immediately prior to reading Successful Freelancing I read Eric Sink's The Business of Software, which is very detailed about a few aspects specifically related to running a small software business. Successful Freelancing covers a wider range of topics, and it is not aimed specifically at software freelancers. If anything, it's aimed more at web designers who probably like nicer typography.

To conclude, The Principles of Successful Freelancing is a great first introduction to the idea of freelancing. It covers all the bases to help someone evaluate whether or not to go into business for himself.

enjoy,
Charles.

Choosing Java Versions on Mac OS X

2009-07-13T15:31:00.000-07:00

While debugging another manifestation of the "wrong library for groovy webtest" bug recently, I found an email thread that makes reference the java_home command (sadly, I can't find an online manual page to link to) to cleanly select a specific version of the JVM under OS X. Here I thought manually pawing through /System/Library/Frameworks to look for versions to set JAVA_HOME to was the "right" way to do it. Learn something new every day.

enjoy,
Charles.

Dealing with DreamHost Server Migration

2009-07-03T13:59:00.000-07:00

DreamHost sent me an email telling me that my account had been moved to another server. OK, fine - "what's that mean to me, Al Franken?" They said that it shouldn't affect most sites, but that you'd have to look out for paths in your applications that look like "/home/.something/username".
So, when when on my clients emailed me to say that his WordPress site was giving him an error message that read:

Unable to create directory /home/.spuds/user/domain.com/wp-content/uploads/2009/07. Is its parent directory writable by the server?

I knew that I had been bitten by this error. OK, how to fix it? I grep'ed the WP PHP files, but couldn't find a path like that. So where was it coming from?

On a hunch, I decided to look at the wp_options table in the WP database and found a row called upload_path, and sure enough, it contained the offending directory. I just removed the ".spuds" portion of the path, as per DH's directions, and it all worked.

So, if you're on DreamHost, and they moved your WordPress blog, and it suddenly stopped working with that error message, try looking through wp_options in your databse. (FYI, DH puts a random string between the wp and the options to prevent collisions with other users in the same database.)

enjoy,
Charles.

Aliens have taken control of my PowerPoint!

2009-06-30T14:50:00.000-07:00

I just cut-and-pasted some text from a PDF into PowerPoint, and this is how it rendered:

The aliens are the bullet points from the source PDF. WTF?

Little Summary of the Open Source Bridge Conference

2009-06-26T15:33:00.000-07:00

It's been about a week since the Open Source Bridge Conference, and here are some brief thoughts about it. I'd summarize it by saying that it's approximately 80% as good as OSCON for nearly one tenth the cost, and it was here in Portland.

Of course, it was smaller than OSCON - a lot smaller. But it had a lot of what I go to OSCON for - talks about cool open-source software. There weren't as many talks, not as many projects, not as many "rock stars", but it's the same basic idea. The exhibitors area made a Soviet-era consumer electronics show look like NES. I felt sorry for them.

As I hinted at, the price (something like $250 for non-early bird) was beyond reasonable - it's cheap, in a good way. At the moment, I'm between clients, so there's no way I could afford over $1000 for OSCON. If I had that kind of money this year, I probably would have blown it on the Java Posse Roundup - maybe next year.

Finally, having it in Portland was crucial for me, especially this year. However, I realized a down side to that: since I'm basically local, at the end of the day, I didn't hang around for social events or the hackers lounge, but I rather went out with friends whom I was staying with in the area. So the fact that it's local sort of limited my participation.

Anyway, it was a great version 1.0, and I look forward to going to OSB again next year. Oh yeah, props out to O'Reilly for being a sponsor of a "competing" conference. That was cool.

Charles.

Version control: Autopia vs. off-roading

2009-06-26T10:39:00.000-07:00

When I was a kid, it was cool to go to Disneyland and ride the Autopia ride. For those of you unfamiliar with it, you get to "drive" a car along a track. As I recall, you get a gas pedal, and the steering worked for about plus-or-minus a foot off the center-line of the track. It is a far cry from driving a real go-cart let alone a real car on the highway or going off-roading (not that I've ever intentionally been off-roading.) Comparing traditional centralized version control systems to distributed version control systems is a bit like autopia vs. off-roading.

With a centralized version control system (VCS), your options are limited, and to a certain extent that can be a good thing. A critical difference between Autopia and a centralized VCS, is that the VCS actually goes somewhere - your code line continues to move forward and doesn't loop back to the starting point.

A distributed VCS (DVCS) can follow the exact same path that a cetralized VCS provides/requires - just like you could drive a 4x4 along the Autopia track. With a DVCS tool like Mercurial, you can implement policies that end up following the same trajectory as you would with Subversion. You have one "master" repository; everyone checks out from it; everyone commits directly to the central repo; the only branching is officially sanctioned, and it occurs in master repo. And that's fine for a lot of applications. Also, some organizations want to operate on the straight and narrow, which is fine if it works for them.

As I mentioned before, I think Mercurial is very simple to use on that path. But, the blessing or curse or fun or power of a DVCS is that you need not follow that track. You can go off the track. Doing so might be the fastest way to get where you're going. Or, you might end up in the weeds. And, you can drive back onto the main road. Developers can head off in some strange direction in their private repo, share their changes with other repos, bring in changes from other repos, abandon their work, or merge it all back into the designated "main" repo. All the while, they have a real VCS tracking their work and saving it - not just random hacking in a random directory.

If Autopia is where you want to be, more power to you. I wouldn't recommend using a DVCS in that environment. Although I don't go seriously "off-road" in my development, I like to have the option to get a little mud on the tires. And even if I'm "driving on the track" and using Mercurial like a centralized VCS, I like to clone a tree onto my laptop to operate disconnected for an extended period, which is something that's not generally possible with traditional VCSs.

My most "extreme" use of Mercurial to-date has been to mirror a P4 repository that I only have intermittent access to. Hg lets me do all my work without having to get the admin to let me onto the P4 repo from by dyanamic IP address. Before that, I would work for weeks without checking in, which makes me nervous. Due to some ignorance and poor planning, I created a bit of a mess in my hg development repo, but I was able straighten it all out, build a series of mq patches, stage them on another repo, and push them back to the P4 tree.

404 Errors with Django flatpages

2009-06-01T16:07:00.001-07:00

In a Python class I'm teaching, I had the students do the Django tutorial with some added bits. One of the added bits was to add an About page using Django flatpages. A couple of students had problems where they were getting 404 errors even though there was a flat page defined at the /about/ URL.

I had one student zip up his project (including his sqlite database) and send it to me. (One of the other "added bits" was that they had to create portable projects - no hardcoded paths in settings.py.) When I first ran his project, sure enough - 404. I poked around, and everything looked good. Then, I used manage.py loaddata to load my data (including a flat page) into his DB, and it all worked fine. So, it was a data issue, not a project-level mis-configuration.

After reading a comment from another student about this problem, I looked into the database using SQLite Manager, where I could see that the flat page was assigned with site_id #3, not the default #1, and somewhere along the line he had created multiple sites in his DB. I looked in settings.py, and sure enough, SITE_ID referred to site #1, not #3 As noted in the flat pages docs, this needs to match (under some circumstance). Changing that, fixed it all up with the students original data.

If you have been having this problem, I am hopeful that the Google sent you here, and you can see how to fix it.

enjoy,
Charles.

How to Get a Software Job w/o Experience.

2009-04-10T14:51:00.000-07:00

This is a talk I gave at WOU last week. I also presented it later the same week at Beaver Bar Camp. In an ideal world, I would have delivered the talk at least a few weeks before the deadline for Google Summer of Code applications. :-( OTOH, maybe some students will join some F/OSS projects this summer, and they can nail the application next year.

How to Get a Software Job w/o Experience

View more presentations from Charles Anderson.

Slideshare liked it (but I'm not sure what their criteria are): "Your presentation How to Get a Software Job w/o Experience is currently being showcased on the 'Career' page by our editorial team."

enjoy,
Charles.

Book: Writing for Scholars

2009-03-24T20:37:00.000-07:00

Writing for Scholars: A Practical Guide to Making Sense and Being Heard by Lynn P. Nygaard
ISBN: 9788215013916

Writing for Scholars is a great guide for (aspiring) academic writers. The simplest thing I could say is that it ought to be required reading for anyone in graduate school who will be doing academic writing - e.g., journal articles or a dissertation. In my experience, academic writing was something that a graduate student was expected to either know already or absorb quickly without little or no coaching.

I've read a couple of books on the subject of academic writing, especially in the area of the sciences. Those books focused on a lot of the minutia of presenting and formatting one's work in a journal or similar medium. Ms. Nygaard takes a much larger view of the writing process, and she de-emphasizes (without completely dismissing) the technical minutia, putting it in the later chapters. She begins by talking about how to develop good writing habits, which is applicable to non-academic writers, too. She also explains the academic dialogue and how an academic paper has to fit into and extend that dialogue.

She continues by explaining how to identify your audience, which is also applicable to non-academic writing. Then she gets down to what I would term the core of the writing process: forming your argument and expressing it in standard academic form (abstract, introduction, method, results, discussion). She also explains how and when to use figures and tables.

A couple of topics that I don't recall reading in other books are: feedback (giving and receiving) and presenting a paper at a conference. Again, both of these are subjects which were never taught in my graduate schooling. These are both crucial topics that complete the academic dialogue.

Throughout the book, Ms. Nygaard includes numerous (sometimes humorous) examples drawn from a wide range of academic disciplines. Perhaps I'm just reading it through my own science-tinted glasses, but I'd say the book does lean more towards the "hard" sciences rather than social sciences or liberal arts. However, I would assume that non-science writers would find this book just as useful as the geeks in the world.

If I had to make a minor criticism of this book, I'd say that Ms. Nygaard should include some references to other sources relating to the various topics she addresses. This is a short book (less than 200 pages), and that's a good thing. But, as a short book, it cannot possibly be the end-all and be-all encyclopedia for academic writing. For example, her chapter on figures and tables is a great introduction, but references to authors like Tufte would serve the (novice) reader well.

In conclusion, Writing for Scholars is a great guide to academic writing. It is a must-read for anyone beginning a career that will involve such writing, and even seasoned writers can learn a few things by filling in some gaps that were left over from learning by osmosis.

enjoy,
Charles.

Book: Pro Django

2009-03-04T15:47:00.000-08:00

Pro Django by Marty Alchin
ISBN: 978-1-4302-1047-4

Pro Django is an excellent book on Django, but it's not for beginners. The term "Pro" gets thrown around a lot, and it gets applied to things that might better be described with "Dummies." This is the Real Mc Coy - it's serious advanced stuff.

The chapters are centered around nice little chunks of the Django system: Models, Views, Forms, Templates, etc. Each chapter is a nice, self-contained bit of Django knowledge, except for Chapter 2, which is a great survey of advanced Python like meta classes. Most chapters also include an Applied Techniques section which gives some examples of how to apply the material in the chapter.

While reading this book, what struck me was how the chapters seem to pack in a level of detail that you'd typically find only in a comprehensive reference, but yet this book is not a bunch of dry reference material, or worse yet, copies of online manuals. The reader gets serious detailed information, but it almost reads like a fluffy tutorial. It's pretty remarkable.

Something that's unique about this book at this time (Q1 2009) is that it covers the 1.0 version of Django. A bunch of the first books on Django were written against 0.96 or earlier. You'd think there wouldn't be much difference (0.04 versions if you only look at the numbers), but the jump to 1.0 was significant for Django. It's nice to have a book that reflects the 1.0 world.

enjoy,
Charles.

Does OpenOffice Base suck?

2009-03-02T15:07:00.000-08:00

Is it just me, or does OpenOffice Base (the database tool in their suite) suck? Or maybe it just sucks on the Mac? Honestly, I'm not throwing bombs just to be a tool. If I'm doing something wrong, please tell me.

For some time I've been looking for a database tool along the lines of Access, but which runs on the Mac and possibly other platforms. It's not that I'm a big Access fan - I've only used it a time or two, but every now and then, I have a task that cries out to be implemented with a database. Creating a desktop application from scratch using something like Swing and JavaDB/Derby seems like overkill (not too mention, way too much work), but I've always thought that there should be a database tool that's based on Java (cross-platform) and some open-source database (e.g., Derby or SQLite).

On paper, Base is just that tool. I heard somewhere that it's written in Java, and I know that it uses HSQLDB, but can use any JDBC database. Looking through the interface it's got tables, forms, and reports. And OpenOffice runs on Macs, Windows, and Linux. Sounds perfect.

Here's just a short list of the issues I've had:

The first time I ran it, it crashed before I'd even defined any tables. D'oh!
Once, the UI locked up (no visible updates but the mouse still worked) while I was trying to add a List Box. I kept trying until it crashed. When it came back, I had a zillion List Boxes in the spot where I was adding them.
After removing the List Boxes, it crashed again. After restarting, the recovery process restored all those list boxes. I removed them all again, it crashed again, and they were all restored again.
Eventually, I removed the List Boxes, saved and quit. After restarting, the boxes were finally gone. This marked a new work pattern - save and quit every ~5 minutes. Sometimes, saving alone wasn't enough to prevent work from being lost.
The replace form control function crashed consistently enough for me to realize it doesn't work. This is a shame because the form wizard creates text boxes by default, which need to be converted. I had to add new controls, wire them up, remove the original text box, and move the new control into place - all while saving frequently.
It took me forever to get a List Box that wasn't tied to a database table or query - e.g., Gender can only be Male or Female (or Gelded on our farm). To do that, you have to turn off wizard mode.
There are various UI boogers on the Mac - e.g., list boxes are sometimes not quite tall enough so the text in the box is chopped when the list is not dropped down.
The documentation that I could find was minimal, at best. I realize that's a common complaint about open-source projects, but for something as big as OpenOffice, I expected more.

Anyway, it's so unusable, I don't think I can use it for my own personal purposes, let alone recommend it to clients. Quite a shame. I'm thinking to checking out FileMaker. It runs on both PCs and Macs, and it has a free 30 day trial.

Charles.

Talk: How To Protect Yourself and Your Computer Online

2009-02-25T14:48:00.001-08:00

This is a talk I recently gave at the Monmouth Senior Center. Nothing earth-shattering for techies, but useful information for the non-techies in town.

How To Protect Yourself and Your Computer Online

View more presentations from Charles Anderson. (tags: securityonline fraud)

Cell Phone (EVDO) for Rural Internet Access

2009-02-20T21:51:00.000-08:00

The name Western Skies (name of this blog and my consulting company) comes from song "Western Skies" by the late Chris LeDoux where he talks about living out west (Wyoming) rather than in Nashville. Similarly, I choose to live out in the country in Oregon instead of Silicon Valley. On the plus side, I have "watch(ed) an eagle fly" (in town, no less) and listened to "the coyotes call at night." On the minus side, there is no broadband Internet access out here. I've recently switched to Verizon EVDO service, and it's mostly working. This is the first of a series of posts on the subject of using EVDO for our sole Internet service.

Previously, I was using 128Kbit (16 Kbyte) per second ISDN. It cost $85/month, which means it was slow and expensive, but the data amount was unlimited. Also, it was a business service from Qwest, which meant when it went out, they were Johnny on the spot to fix it - they even asked at 9pm if it was OK if they didn't roll a truck until morning or if I needed it fixed that night.

I had been thinking of switching for some time, but Qwest forced my hand when they notified me that they would no longer serve as an ISP for the ISDN line. They still provide ISDN service, but their ISP (Qwest.net) no longer accepts ISDN calls. To add insult to injury, they suggested that I see if DSL was available - if it were, there's no way I'd suffer the indignity of ISDN! I could have changed ISPs, but the ones I found all charged more and/or had usage limits expressed in hours per month (not MB/GB, just hours of connect time).

Since there's no DSL or Cable, the only other contender would be satellite. However, at times, I do a lot of work at the command line via ssh, and the latency of satellite would have been completely impractical. A neighbor of ours has it, and she says she can't even run IM over it very well.

When we first moved out here, there wasn't hardly any cell phone coverage. Mostly our phones just spent their batteries looking for signal. Then, Verizon put up a tower that we can see from our driveway - a pretty clear line-of-sight. Our Sprint cell phones get 4 bars of (roaming) signal, except when they catch wind of 1 bar of Sprint, which they'll chase blindly.

After checking out Verizon's offerings online and talking to a sales rep, I signed up for service and "bought" a UM 175 EVDO modem (free with 2 year contract) at Costco (to save on the activation fee). Verizon offers a 30 day trial, which I figured I'd use to test things out before cancelling the ISDN line. Even though our phones got a strong signal, I wasn't certain that the data signal would work. Of course, you'd think that if a telco bothered to put up a tower (or an antenna on someone's leased tower), they'd provision it with all the latest features (e.g., EVDO), but then again telcos don't always behave rationally.

To make a long story short, it really does seem to work. The bandwidth is much better - I see anywhere from 100KB/sec to 800KBps with 100-300 KBps being common. Not great compared to the fiber in town, but much better than 16KBps. Ping shows the latency being pretty large (150-250ms), but it seemed alright when I tried it over ssh. (I'm no longer working for the client where I was using ssh full-time, so I haven't really put it to a real test.)

However, and this could be huge, the 5GB/month cap is problematic. In theory, our regular email and web surfing fits well within that limit, however that doesn't leave much room for podcasts or video (which I never got into on ISDN, anyway.) The cap works out to ~170MB per day, and many podcast episodes are ~50MB. So, I'm still trying to figure out how to live with the cap - giving up podcasts is not an option.

At some point, I'll post about my experiences setting up the Verizon UM 175 USB modem on a couple of Macs, as well as my new found hobby: Podcast Mule - downloading podcasts (and OS updates) from WiFi access points in town and bringing them home on my laptop.

Charles.

Speaking on Groovy again

2009-01-30T19:02:00.000-08:00

I'll be repeating my talk on Groovy at noon on February 3rd at noon in Salem, Oregon. From the email notice:

This presentation will be in Information Systems Division conference room on the first floor, suite 103, of the State Public Service Building. Here is a Google Map link for the location.

Parking on Court and Capitol streets is the closest to the Public Service Building. The spots are metered.

Maybe I'll see you there,
Charles.

P.S. My slide deck will be almost identical to the last Groovy talk, so I won't be posting a new one.

Converting a WordPress blog to a 'One-Click Install' on DreamHost

2009-01-30T18:38:00.000-08:00

We set up a WordPress blog on DreamHost a while ago for a client. It was a custom install due to a bunch of craziness related to him getting booted from his old hosting setup. It all worked fine, and then DreamHost sent me a "nag email" (their term) asking me to upgrade because the old version had security issues. Fair enough.

I had another site to upgrade, but it had been installed as a one-click install. The upgrade for that was one click (plus some backing up). I could have manually upgraded this custom install, but if I did that, I knew I'd have to continue doing manual upgrades. Therefore, I wanted to convert this custom WP install to a one-click install.

DreamHost didn't seem to have any instructions, but how hard could it be? Basically, the procedure was pretty similar to what you'd go through to move a WP from one server to another.

Backup the database and the old site: I backed up the old DB using mysqldump, and I tarred up the old site.
Export the old site: from the WP admin page, select Export and dump all authors.
Create a new database in the DreamHost control panel. In theory, you could drop all the tables in the old DB and re-use it, but I was being paranoid, which paided off later.
Move old site out of the way: just move the old site directory to a new name - e.g., mv site.com save.site.com.
Create a new 'One-Click Install' WordPress site using Advanced Mode, since we had a custom theme and a bunch of other content (pictures). This was put in the new database. After DreamHost emailed me to tell me that was done, I logged in and changed the admin password.
Import the old content from the Admin Import function. Actually, DreamHost's email included the URL for that page. One issue I ran into was the "user name" for the old posts: WP thought the old posts by admin were from some other user because in the old blog, the admin user had a different display name.
Copy in the custom theme and other content from the old saved directory into the new blog site. Enable the theme. Set up Akismet - this is where having the old DB around was useful; I just looked in the wp_options table to find the API key instead of having to dig around for it.

enjoy,
Charles.

Jython talk at SJUG

2009-01-22T15:45:00.001-08:00

Here are the slides of a talk I gave last February at the Salem Java Users Group - SJUG. Although it looks like it's primarily about Python and Jython, my bigger emphasis was on extension programming - scripting existing Java code. In other words, a form of polyglot programming.

Jython: Integrating Python and Java

View more presentations or upload your own. (tags: jython python)

Enjoy,
Charles.

Book: Java Power Tools

2009-01-21T21:09:00.000-08:00

Java Power Tools by John Ferguson Smart
ISBN: 978-0-596-52793-8

Java Power Tools provides a fairly detailed introduction to a number of tools for Java programmers. It fits nicely between the O'Reilly Hacks series and having a dozen books like Ant: The Definitive Guide, 2nd Edition. Like the Hacks books, Java Power Tools provides an introduction to a bunch of tools. The Hacks books are great for answering the question "I've heard of that tool, but where does it fit?" But whereas the Hacks books provide just an appetizer, this book provides a main course, enough to get seriously started with the tool being discussed. And then, if you want all the gory details, a Definitive Guide could provide the full five-course meal.

The selection of tools presented was really good, at least for me. For example, I know about continuous integrations servers, but I haven't set one up. At one client site, they were using Hudson, which I had some exposure to, but didn't know much about the others like Cruise Control, Continuum, and LuntBuild. Similarly, I've been using JUnit 3.x for years, but I didn't really know what was different in JUnit 4 or how that compares to TestNG. This book provided me with a great overview of these and other tools. Java Power Tools provides a great way to get up to speed with a general area of tooling (e.g., continuous integration servers) or a good cross-section of the majority of the Java tools in use today.

If I had to pick something to complain about, it would be Part II - Version Control Tools. These aren't really Java tools, although every programmer (Java or otherwise) should be using them. Or given the decision to include version control tools, I'd suggest excluding CVS because it's old and including at least one distributed version control tool like Mercurial (used by the Open JDK project and NetBeans) or git (used by the Linux kernel).

So, in conclusion, unless you have no free will about tool selection or you already know all of these tools backwards and forwards, I highly recommend this book to almost any Java programmer.

enjoy,
Charles.

Groovy at SJUG

2009-01-07T15:33:00.000-08:00

On January 6th, I spoke at the Salem Java Users Group on Groovy. The premise was not to replace Java, but rather to show how it can be used in addition to Java. Here are the slides.

Groovy a Scripting Language for Java

View more presentations or upload your own. (tags: java. groovy)

Enjoy,
Charles.

Book: The Productive Programmer

2008-11-26T14:59:00.000-08:00

The Productive Programmer by Neil Ford
ISBN: 978-0-596-51978-0

I've always had a more-than-passing interest in productivity porn, as Merlin Mann calls it. I'm not obsessive about seeking out productivity books (I can quit any time), but I've read more than a few such books in my day, and I do keep my tasks organized with OmniFocus. When I saw The Productive Programmer at the Powell's table at OSCON 2008, I confess I got pretty tingly, kinda like "when we used to climb the rope in gym class." Here was a book talking about productivity aimed specifically at what I do every day - programming, not management, not sales, not coaching a football team, but programming. In the words of Cartman, "sweet."

That said, I wouldn't classify this book as hard-core productivity porn. It doesn't lay out a dogmatic formula, nor does it suggest or require specific tools or techniques. And that's probably a good thing; in my experience, programmers can be some of the most opinionated people I've known, especially when it comes to their craft (e.g., editor wars). If the author had attempted to prescribe a specific set of practices, I think almost every programmer would have found something to hate about the book, and what's the point of that - we could just as easily go back to ragging on emacs, Windows, or Steve Jobs.

Instead, Mr. Ford offers a number of possible suggestions that one can take or leave. These are organized into two parts: mechanics (the productivity principles) and practice (philosphy), or what I would call tactical (little picture) and strategic (big picture) techniques. Mechanics includes things like controlling interruptions from things like email and using tools like Quicksilver (which I finally started using after reading this book). Philosophy includes things like test-driven development/design (TDD) and using things like static analysis tools.

The various suggestions were all very well and good, but what I liked most about this book is that it made me thing about mom-and-apple-pie topics like TDD (of course, we all write unit tests, right?) from a totally different angle - productivity. Of course, that what the agile folks have been saying all along, but somehow this book shed a whole new light on it and helps drive it even deeper. And the whole book got me thinking about the bigger question - "how can I be more productive and effective in my programming?"

Like I said, this isn't hard-core productivity porn, but it's a very useful and approachable guide to productivity by a programmer for programmers. Maybe that makes it productivity literary erotica?

enjoy,
Charles.

In Praise of TDD and Mocking

2008-11-26T14:33:00.000-08:00

As noted elsewhere in the blog, I've been doing a bunch of work to script ESXi using Vmware's (sometimes troublesome) RCLI tools. I'm developing a higher-level set of code written in Python. (In theory, using Vmware's Perl toolkit would be cleaner, and I wouldn't have to bitch about the RCLI tools, but I've done enough Perl for one lifetime.) In addition to the mom-and-apple-pie goodness of TDD, I'm also getting a huge productivity boost by employing TDD via mocking.

In the code I'm writing, each method typically makes one or more RCLI calls. Each RCLI call takes 3-5 seconds. I structured the code so that the RCLI invocations all funnel through one point in the code which can easily be monkey-patched to go through a mock function. (More details on that in a later post.) After mocking, the result is that I can execute 20 tests with dozens of mocked RCLI calls in less than a tenth of second. After the unit tests pass, I can push the code out to a real host and run real RCLI comannds, and for the most part, "it just works."

When I started, I figured mocking was the only way to unit test this code, and it was more convenient to develop the code on a machine where I don't have the RCLI installed. The performance boost was unexpected but by far the most significant benefit of the unit tests. And when I needed to perform a couple of refactorings during the development, I got to enjoy wicked fast speed plus safety - two great tastes that taste great together.

Charles.

Vmware RCLI and exit status

2008-11-20T12:20:00.000-08:00

Yet another gripe about Vmware's RCLI commands: they almost always return an exit status of zero, even if the command failed. For example, 'vmware-cmd -s unregister' for a non-existent virtual machine, return a status of zero (which is the same as if the command succeeded). One has to look at the standard output of the command to see "No virtual machine found." This is OK for an interactive user, but it's a pain-in-the-ass if you're trying to write scripts, as I happen to be. For every command, I have to parse the output to see if it succeed or not.

But then some commands (e.g., vifs) do return useful exit codes, at least some of the time. This whole process has to be handled on a case-by-case basis.

Speaking of parsing output, the outputs are not consistent. For example, when that unregister command is successful, it returns "unregister() = 1". Note there is a space on both sides of the equals sign. When the corresponding register command succeeds, it returns "register() =1". Note that there is no space on the left side of the equal sign. As I write code to parse these outputs (because I can't count on the exit status), I can't help but wonder how brittle this code will be...

Charles.

Book: Working Effectively with Legacy Code

2008-11-07T21:02:00.000-08:00

Working Effectively with Legacy Code by Michael Feathers
ISBN: 0-13-11705-2

"...legacy code is simply code without tests... Code without tests is bad code."
From those statements, it doesn't take much to figure out what this book is about - how to write unit tests for code without tests. Of course, if you've ever tried to do that, you know that it's easier said than done. Many/most programmers who inherit a big ball of mud of code without tests (legacy code) just punt; the existing code has no tests, I can't see how to get any of it under test, so I'll just hack and pray - just like the original author(s) did.

That's where this book comes in. It's a primarily large collection of recipes about how to write unit tests for legacy code. That said, the focus is not really on how to write the tests but rather how to get chunks of the legacy code into a test harness so that you can write unit tests to characterize the existing functionality before adding or modifying functionality. It also contains techniques to add new functionality in such a way that you can test it immediately and possibly execute the "clean up" that the original author(s) promised would happen as soon as that next deadline was reached - all those years and deadlines ago.

The bulk of the book (Part 2) is organized as a series of complaints or excuses and how to deal with them. These include such topics as "My application has no structure," "Dependencies on libraries are killing me," and "This class is too big, and I don't want it to get any bigger." In each chapter, the author provides examples (in multiple languages - Java, C++, C, etc.) of these problems and specific techniques that can be used to address them. The last chapter (Part 3) is an encyclopedia of the techniques for easy reference.

If you're lucky enough to do only green-field development, you might think this book would be useless. However, one interpretation of this books is that it is a list of sins to avoid while your playing in the green field. And, many of the techniques can be interpreted as best practices for how to write you code to ensure it's testable. (Of course, you're following test driven development and achieving near 100% coverage, so that would never be a problem with your new code, would it? :-)

My one disappointment with this book was that I was hoping it would provide ideas about how to create higher-level (e.g., functional) tests. Of course, high-level tests are no substitute for unit tests. It's just that I was tasked with creating "some tests" quickly for an entire application, and unit tests are not practical in this particular case, which is my problem, not the author's.

This book is an excellent resource and cookbook for how to add unit tests to an existing code base that lacks tests, and it also provides design and implementation templates to ensure that new code is testable as it's created.

enjoy,
Charles.

vmware-cmd: SystemError=HASH(0x95d8d70)

2008-11-05T15:46:00.000-08:00

More tales from the dark side. Trying to start a VM out on an ESXi host:

vmware-cmd  [conn options]  '[datastore] path-to-vm/conf.vmx' start hard

Returns this output:

Fault string: A general system error occurred: Internal error
Fault detail: SystemError=HASH(0x95d8d70)

I spent hours permuting the parameters. And vmware-cmd's -v option doesn't dump out all of the SOAP guts, so I had no view on what was going on. Finally I ran a vifs --dir out on the VM dir and found a bunch of vmware.log files. I fetched one that showed a normal startup and then this:

Nov 05 05:51:51.883: vmx| [msg.License.product.expired] This product has expired.
Nov 05 05:51:51.883: vmx| Be sure that your host machine's date and time are set correctly.

There was a botched version of ESXi that we installed that contained a time bomb, and it had come home to roost. Fair enough - our bad, but certainly there could be a better error message than SystemError=HASH(0x95d8d70).

Charles.

Vmware ESXi - Unable to clone virtual disk

2008-11-05T11:56:00.000-08:00

To import a VM from VMware Server to ESXi one must convert the disk by cloning the old one using vmkfstools. Fair enough. Doing this using the ESXi RCLI tools should look something like (based on what I've seen for non-RCLI for ESX):
vmkfstools [conn-options] -i old-disk new-disk -d thin

However, when you do this with RCLI, you get the following useless error message:
Unable to clone virtual disk : A general system error occurred: Internal error

Using the --verbose flag to look at the gory SOAP details I could see that the thin option wasn't getting sent. Looking through the Perl code it looks like one needs to specify both the -d and -a (adapter type) options. Once I added -a lsilogic it worked like a charm.

My big complaint (and I have other examples of this up my sleeve) is if the user makes a simple, bonehead parameter error, the command should point it out rather than saying "general system error...internal error." I hope this post will help someone else out if s/he is unlucky enough to encounter this error message.

Charles.

WebTest Groovy example thows NoSuchMethodError

2008-10-30T14:09:00.000-07:00

I've been working on a project to do functional testing on a web application using WebTest. I've coded up some prototypes using the standard XML/Ant format/language/DSL, but I wanted to try doing it in Groovy, since I'm not a big fan of XML. I copied the sample application from the manual, but when I ran it I was getting and exception:

Caught: : java.lang.NoSuchMethodError: org.apache.xpath.compiler.FunctionTable.installFunction(Ljava/lang/String;Ljava/lang/Class;)I
at test.run(test.groovy:33)
at test.main(test.groovy)

(If you're following along at home, you'll note that that line number is too large for the example - I've already hacked some stuff into it.)

I couldn't find any examples online of people having exactly the same problem, but I'd seen other errors blaming an old version of xalan.jar for this problem, especially under Mac OS 10.5 - my platform. (BTW, this happened under both Java5 and Java6.) Although I didn't have CLASSPATH set to anything, I figured I'd try setting it to have the version of xalan.jar that ships with WebTest, and it worked. So, my complete invocation line is:

groovy -cp /usr/local/java/webtest_1720/lib/xalan-2.7.0.jar -Dwebtest.home=/usr/local/java/webtest_1720 test.groovy

enjoy,
Charles.

VMware ESXi default resource pool name

2008-10-29T16:05:00.001-07:00

I've been playing around with ESXi trying to figure out how to use their Remote Command Line Interface (RCLI) to import and run a VM. This has been a major PITA, which I suppose I could have been documenting as I go but I didn't. (Full disclosure - I'm doing this without any formal training, without their snazzy Virtual Infrastructure tools, or anything else.)

According to VMware's RCLI Installation and Reference Guide, the data center and resource pool parameters to vmware-cmd -s register are optional, but nonetheless, it kept telling me "Must specify resource pool".

Eventually, I found a forum thread that contained the answer: Resources. (Kinda like "plastics" only different.) I don't have a data center set up, but I found that using almost any word for the data center name seemed to work.

So, in the end, this is what worked for me (your milage may vary):

vmware-cmd  [conn options] -s register '[datastore-name] vm-dir/conf.vmx' root Resources

enjoy,
Charles.

Increasing Memory for Ant

2008-09-04T16:19:00.000-07:00

I had an issue today with the javac Ant task running out of memory when compiling a project (~800 files) under Java 6 on the Mac. It was fine under the default Java 5 compiler on the Mac, and it was fine under Java 6 on the PC. Initially, I set memoryMaximumSize on the javac task, which also required that fork be set.
This was OK, but since this was specific to compiling on the Mac, I figured the solution should be specific to my environment on the Mac. After looking at the source of the ant script, I realized that the ANT_OPTS variable could be set to contain an option (-Xmx512m) for the JRE running Ant, and that ANT_OPTS could be set from either ~/.antrc or ~/.ant/ant.conf. So, I created ~/.antrc with one line: ANT_OPTS=-DXmx512m.
(Update: oops, that should be ANT_OPTS=-Xmx512m)
And it worked perfectly, of course - so simple it had to work.

enjoy,
Charles.

Ant, JavaDoc, and CreateProcess error=87

2008-08-28T15:12:00.000-07:00

I've been developing an Ant build file for a large-ish base of existing code. I've been using my Mac as the primary devleopment platform, but the ultimate target (for the other developers) is Windows. My task to build the javadocs runs fine on OS X, but the first time I rant it on Windows, I got the following:

c:\demo\build.xml:180: Javadoc failed: java.io.IOException: Cannot run program "c:\Program Files\Java\jdk1.6.0\bin\javadoc.exe": CreateProcess error=87, The parameter is incorrect

After about 10 minutes of Googling, I found a/the simple solution: add the useexternalfile attribute to the javadoc task in Ant. Voila , back in business, on the PC at least, and it didn't seem to break the build on the Mac.

Charles.

Book: Essential SQLAlchemy

2008-08-12T21:02:00.000-07:00

Essential SQLAlchemy by Rick Copeland
O'Reilly Media
ISBN 10: 0-596-51614-2 / ISBN 13: 9780596516147

This is a great book describing how to use SQLAlchemy to connect Python programs to databases. In fact, at the moment (mid-summer 2008), it is the book, since there are no other books on the subject, yet. Athough I am not (yet) a SQLAlchemy user, this book seems to cover all of the core topics in SQLAlchemy. The text includes many straightforward examples of how to use various facilities in SQLAlchemy and how to map various database programming problems into Python code via SQLAlchemy. Copeland also provides a whirlwind tour of some extensions to SQLAlchemy.

I heard about SQLAlchemy project on the This Week in Django podcast. Django doesn’t use SQLAlchmey, but it does use a similar object-relational mapper (ORM). As I mentioned, I haven’t used SQLAlchemy so I came into this book with a somewhat blank slate. I have, however, been programming in Python since before 1.0, and I’ve worked with database APIs and ORMs since the early 90s in C++, Java, and Python. So, I was familiar with the basic landscape of database programming, even if I hadn’t used SQLAlchemy. And, I’m currently working on a large Python project that is coded using the Python database API directly, which is very tedious. So, the whole time I was reading this book, I was looking at how to fit SQLAlchemy into this existing code base.

To be honest, the first chapter (the proverbial introduction) almost turned me off. The author starts out slowly enough, but then he starts touching on a huge number details, which were glazing my eyes over. However, the second chapter (getting started) started back at ground zero and stepped through everything in a nice clear fashion, and the rest of the book continued in that vein. He covers all the topics you would expect in a database programming book: queries, updates, joins, the built-in types, and how to hook in to provide support for your own types.

Something I didn’t realize about SQLAlchemy coming into this is that SQLAlchemy is both an ORM (what I expected) as well as a high-level, database-independent API. Which is to say, you can just access the database as tables, columns and rows rather than as classes, attributes, and object instances. Although I’d personally prefer to use the ORM, I can imagine cases where it might not be the right tool for the job, and it’s good to have a choice.

I was also surprised to see the ORM supports two styles of object-relational access: the data mapper pattern (which I had seen in Django and Hibernate) and the active record (used in Ruby). The author does a good job of explaining both of these and how to use them. He even devotes a whole chapter to Exlir, which is an extension that implements the active record pattern.

One thing that many people might consider odd is the fact that although SQLAlchemy is an ORM, the author waits until chapter eight to discuss how to map object inheritance hierarchies onto relational databases. Most books I’ve read on ORMs discuss this topic early, but I applaud Copeland’s decisions to hold off on discussing it. When books bring this up early (e.g., in chapter three), the discussion often gets bogged down in details, which glaze the reader’s eyes. I’ve dealt with the issue of inheritance mapping enough in ORMs I’ve used and those I’ve written enough that I’m not that interested in the topic (assuming the tool provides the typical, reasonable solutions), and was grateful that he held off on it.

One issue I had with the overall structure of the book is that I’m hard pressed to pigeon-hole the book. Books about a single technology such as SQLAlchemy usually occupy one end or the other of a spectrum. Either they’re hard-core references, often times copying-and-pasting API documentation from a web site (I really hate that), or they’re largish tutorials that may or may not contain enough technical meat. This is a short (roughly 200 page) book that contains plenty of technical meat, but it also includes some simple tutorial motivations for using various capabilities of the tool. Although this mix felt odd to me, I’m sure it will be perfect when I go to apply SQLAlchemy to my existing database projects since I don’t really want a hand-holding tutorial, but a pure reference wouldn’t quite work for me either when I’m just starting out.

In conclusion, Essential SQLAlchemy provides a thorough presentation of the SQLAlchemy tool for interfacing Python code to SQL databases. The author covers a number of different methods in which SQLAlchemy can be used to access databases from Python, and he provides plenty of details of the various APIs available to the programmer.

Enjoy,
Charles.

Converting a Django Site to newforms-admin

2008-07-06T16:16:00.000-07:00

Right after I learned about Django at OSCON 2006, I created a prototype to replace an existing PHP site. Due to various distractions, I haven't done much with that Django site since then and it never went live, but I've come back to it recently.

My first order of business was to upgrade from 0.95 to something modern. The first stop was to upgrade to trunk. Following the instructions on the Django site, I got it running on trunk and only had to change maxlength to max_length in my models.

Since, newforms-admin will be merging with trunk soon, porting to newforms-admin was my next step. I pointed my django install at my newforms admin svn tree and let it rip - i.e., I didn't bother reading any documentation or anything.
It blew up with a nasty-looking traceback that ended with this:

func(self, *args, **kwargs)
TypeError: __init__() got an unexpected keyword argument 'prepopulate_from'

A quick Google search turned up this thread on the Django Users mailing list. Five minutes of reading the newforms-admin wiki page and refactoring my code, and I was up and running. So, despite the ugly-looking traceback, it just wan't that bad.

Granted, my project is pretty small because I never had much time to work on it. However, all in all, the conversion process from 0.95 to the bleeding edge of newforms-admin was pretty painless.

Another happy Django user,
Charles.

Book: Effective Java

2008-07-06T13:06:00.000-07:00

Effective Java, 2nd Edition by Joshua Bloch. ISBN 0321356683

This is the best book that I've read that I didn't know I needed to read. If you are a pretty good Java programmer, and you want to be better, this is a book you should definitely read.

Back in the day (BITD), we had Scott Meyer's book Effective C++, and we needed it. Even without templates and all the "new" things in C++, C++ was a large and complex language. I loved Effective C++ not only because it had lots of tips to keep you out of trouble when coding C++, but also because the organization was brilliant: each tip was about the right length to be read while sitting on the can.

I ignored the first edition of Effective Java book when it came out because I smugly assumed that Java was so superior to C++ (Java is C++ with the pointy bits filed down) that it wasn't really necessary. And that may or may not have been true in the 1.0 or 1.1 days of Java, but with the release of Java 5, the language has certainly grown and is now sufficiently complex that a book like this is a necessity, especially if you've recently moved up to Java 5 or Java 6. Lucky for me, this book landed on my desk against my will.

The book uses that same sitting-on-the-can format from Effective C++; the information is broken down into nice, small bits of information that you can read one or two at a time. These are grouped into 10 sections. There are specific sections for Java 5 features like Generics and Enumerations, which is great for people like me who were stuck on 1.4 for far too long.

What is really amazing about this book is that I learned a lot about topics that I thought I already knew about. For example, I've used serialization in a number of formats (built-in and do-it-yourself) for years, but I realized there's a lot more that I had no idea about. I will certainly think twice before doing any serialization in the future.

Unless you're the sort that knows the Java Language Specification by heart, you need to read this book.

Enjoy,
Charles.

Configuring TortiseMerge for use with Mercurial

2008-06-24T17:42:00.000-07:00

I've been using Mercurial primarily on a Mac, but also on a Windows box for a client. Since the Windows box isn't my primary platform, it tends to be under-configured/under-tricked-out. After my first conflicted hg merge, it was time to set up a merge program.
I guess Mercurial looks through the registry where it was finding some wacked out description for P4 merge. (The client currently uses P4.) It was referring to some path that didn't exist.
I have TortiseSVN on the machine from work for a different client, so I figured I'd use that merge program. Googling around, I didn't find an example, so here you go (even though it's not real hard - see MergeToolConfiguration for details):

[ui]
merge=TortoiseMerge
[merge-tools]
TortoiseMerge.executable=C:\Programs\TortoiseSVN\bin\TortoiseMerge.exe
TortoiseMerge.args=/mine:$local /theirs:$other /base:$base -o /merged:$output

enjoy,
Charles.

DVCS (Mercurial) for Students

2008-05-27T17:10:00.000-07:00

I just had a bit of an epiphany for yet another reason why distributed version control systems (DVCS) like Mercurial rock. In an advanced software engineering class (e.g., a capstone project), it would be appropriate to have project teams using a SCM/VCS tool. At my campus, we've never pushed that because it can be a pain to set up a server for students to access. Gotta have a dedicated host for it. Firewalls have to be open. Permissions and users have to be set up. Yada, yada, yada.

However, with a DVCS tool like Mercurial, it would be trivial and wouldn't require any networking at all. Students could use the modern equivalent of SneakerNet: ThumbNet. Put the whole repo on a thumb drive, meet up with your project partners, push and pull, done. Or, if the project is small enough - just email whole repos around. In this context, "distributed" also means you don't need any support from The Man, and that's a good thing.

For better or worse, I don't teach those classes, but if I did....

Charles.

@Override is your friend

2008-05-16T12:40:00.000-07:00

When Java 5 came out, I had my head down teaching and wasn't really paying attention. I've since started working with it and have been using annotations for "big" tasks like Hibernate/JPA metadata. I was pretty underwhelmed by @Override - one of the only stock annotations. When I implement toString, I know I'm overriding the method on Object. Who cares?

The other day I was beating my head into the desk trying to figure out why a Swing table wasn't editable. I had overridden the isCellEditable method on JTable, but the cells weren't editable. Then, I remembered something from the annoations tutorial I'd read at some point: "While it's not required ... it helps to prevent errors." So, I added @Override, and sure enough - I'd misspelled the method name, just the sort of error that @Override can prevent.

I've got the religion. And like any recent convert, I suggest you get it too.

enjoy,
Charles.

Data Driven, my eye!

2008-05-12T22:06:00.000-07:00

I've started using the WebTest web testing framework. Mostly, it's pretty cool. However, I have a bone to pick with screencast demonstrating the dataDriven task.

There's a "slide" that says "Do you know the dataDriven Ant task?" I know of no such standard Ant task. It turns out that it's specific to WebTest. Not really clear.
They show no configuration steps to use it, implying that it works out-of-the-box. I don't know if my environment is wacked (I installed from the developer build, as they suggested), but I had to add an Ant taskdef referring to com.canoo.ant.task.PropertyTableTask, and I only found that by looking in the source.
The screencast shows running ant at the command line, which is how I've been running my tests, but I had to run their webtest script instead. Again, maybe my installation is wacked.
I really wish it could handle data in a TSV/CSV/text file, since I don't have Excel installed on the machine where I'm running these tests, but it only seems to accept an xls file.
Just to add insult to injury, Google Spreadsheet (which I'm using to generate the data file) seems to append a bunch of empty lines to my spreadsheet, which causes the dataDriven task to repeat the last line 90-odd times.

Grrr,
Charles - aka Cranky Pants.

Update:
OK, so it sucked to be me, but not any more. I figured out my various issues with the dataDriven task. It turns out that the screencast (clearly) shows them developing in the tests directory of a WebTest project. I missed that and tried to use dataDriven in the top-level build.xml in a target that didn't declare wt.defineTasks as a dependency. Guess what wt.defineTask does - yup, it does the taskdef.
I coupled that breakthrough with breaking down and using Excel to create the xls file (instead of exporting from Google) and viola - I'm livin' large and no longer cranky.

enjoy,
Charles.

P.S. Apologies to Cheech and Chong.

Universal???

2008-05-07T11:50:00.000-07:00

Apple has released Java6 for the Mac, which I have been eagerly awaiting. However, their download page is a bit contradictory:

This is limited to 64-bit Intel macs (which is fine with me), but yet they still call it Universal. What's Universal about that? The only way I could imagine it being less universal is if it they specified a number of cores or CPU speed.

Anyway, Java6 is good stuff for those of us on Universal Core2 Duo iMacs.

Charles.

Netbeans is a memory hog?

2008-03-29T14:07:00.000-07:00

I was looking at Activity Monitor on my iMac, and I couldn't read the size entry for the Netbeans process that was running. So, I selected it and got a detail report:

I knew that Java and Netbeans could be memory hogs, but 16 million TB (16 exabytes) is a bit much. Good thing I have virtual memory! (I didn't alter the image, but clearly it's a bug with Activity Monitor and/or OS X 10.5.2.)

Charles.

Simple, complete example of Python getstate and setstate

2008-02-28T23:03:00.000-08:00

I've been doing serialization and/or object-relation mapping in languages like C++ and Java for at least 15 years. I've known about Python's serialization facility (the pickle and cPickle modules) for as long as they've existed, but I've never had a need to use them. Recently, I needed to pickle an object to store in memcached to reduce database traffic.

Wouldn't you know it - the first class I try to pickle throws an exception because it contains some attributes that can't be serialized. I couldn't figure out where the problem was because the Exception and trace back didn't include the name of the attribute that contained the threading lock that couldn't be serialized. However, a quick look at the code revealed a couple of suspects.

Even though I didn't know which attributes were causing the problem, I knew that the only solution would be to take control of the serialization process. Once I could pick and choose which attributes were being pickled, I could search for the offender(s). As it turned out, both of my initial suspects were guilty of evading pickling.

From the documentation on pickling, I could see that implementing the __getstate__ and __setstate__ methods, but it wasn't clear what those methods need to look like. I found an example online, but the guy was having problems (it was posted to a mailing list), and as I implemented my own methods, I realized what his problem was. So, here's the code:


    def __getstate__(self):
        result = self.__dict__.copy()
        del result['log']
        del result['cfg']
        return result

The problem I was having with pickling were the logging and configuration attributes. These needed to be removed from the object before pickling. Fortunately, they're not unique to the instance, so they're easy to recreate during unpickling.

As you can tell, __getstate__ returns a dictionary of the object's state. By default (if you didn't implement the method), this is just the __dict__ member. To exclude some attributes, we just need to delete the keys from the dictionary. However, the crucial step is that you have to make a (shallow) copy of __dict__ first. Otherwise, deleting the keys from the dictionary is the same as deleting the attributes from the instance, which would be bad. (This is where the other example I found online failed - he didn't make a copy.)

The __setstate__ method is the reverse, only we don't have to mess with copies:


def __setstate__(self, dict):
    self.__dict__ = dict
    cfg = self.cfg = getConfig()
    self.log = getLog()

Enjoy,
Charles.

Mac OS X 10.5.2 did not completely fix stacks

2008-02-23T23:11:00.000-08:00

Apple's newest update to Leopard, 10.5.2, has greatly improved the new Stack feature by adding a hierarchical list view, but it is still not as functional as the list view in Tiger. As I noted before, I created my own directory that has a collection of aliases (symbolic links) to the applications I use most frequently, as well as links to Applications, Utilities, and the LocalApps directory where I install third-party applications. The problem is even with the 10.5.2 update, the list view does not follow the symbolic links, so my folder of links is basically useless.

I'm still pleased with the list view - it is a huge improvement, but I won't be totally satisfied until it follows links.

Charles.

Simply Mercurial

2008-02-17T01:32:00.000-08:00

Mercurial is the easiest revision control system I've used, "and so can you" to quote Stephen Colbert. I became interested in the idea of a distributed SCM tool in order to keep my revision history with me while I'm on the road and not necessarily connected. I would have assumed that to get that power, the tool would be more complex - you can't get something for free, right? However, Mercurial is so easy to use, I'm using it for simple one-off revision needs.

Consider the case of a lone developer with a modest number of files to keep track of. To use Mercurial, all he needs to do change into the directory where the files are and run:hg init
That creates a repository, hidden in the .hg subdirectory, and sets the directory up as a working directory. The hg status command shows that none of the files is under control, yet. Running hg add * (or whatever subset of the files is appropriate) marks all of the files to be added to the repository. Finally, hg commit commits the files.

The real beauty was in that first step - hg init. That is so much easier than CVS or Subversion where you either have to create a new repository or figure out where in an existing repository you want to put these files. And it's easier than the dinosaurs, RCS and SCCS, where you have to set up subdirectories to hold the version files in every subdirectory - not to mention the fact that those tools don't really deal with multiple users.

Mercurial is about as simple as can be, and if you never work with multiple developers and passing changes around between developers and repositories, then it stays that simple. Period.

enjoy,
Charles.

Complete example of __getattr_ in Python

2008-02-02T23:34:00.000-08:00

I've always known about the _getattr_ function on classes in Python and how it could theoretically be used. However, I never had a real need to implement it, and so I had never actually implemented __getattr__. For whatever reason, it was a tad more difficult that I thought, so I figured I'd share an example with you'all.

In case you don't already know, the idea is that any time any code makes are reference to an attribute a class (e.g., obj.x ), __getattr__ gets called to fetch or compute the value of the attribute. This function can do almost anything, but you must be careful when making references to attributes, because that will trigger a recursive call to __getattr__.

In my case, I was writing some code for unit testing. I needed to create a mock object that's used to store configuration information for the system. In the real object, every configuration attribute is initialized from an ini file parsed by ConfigParser in the constructor. For testing, didn't want to have a huge configuration file for every test. So, I wanted to create a system that performed lazy initialization of the data attributes - i.e., only look in the ini file if we actually need a given item, and if the attribute is never referenced, we never need to fetch it from the ini file. Therefore, the ini file only needs the attributes that are actually used by a given test. Implementing __getattr__ is the way to hook into the process to provide this lazy initialization.

The basic outline/algorithm is:

If the attribute already exists on self, return that value
Fetch/compute the missing value
Store the value on self for subsequent use
Return the value

The key to making this work and avoiding infinite recursion is the __dict__ attribute, which is a (regular) dictionary, the keys of which are attributes that exist on the object and the values are the values of the attributes. We can access these keys and values without going through __getattr__, thus avoiding recursion.


def __getattr__(self, attrName):
  if not self.__dict__.has_key(attrName):
     value = self.fetchAttr(attrName)    # computes the value
     self.__dict__[attrName] = value
  return self.__dict__[attrName]

It's pretty straightforward. In retrospect, I'm not sure what tripped me up when I first went to implement it. In the end, the fetchAttr function ended up being pretty fancy, but I'll write more about that later. You gotta love a dynamic language like Python that makes this as simple as it is, even if it does require a bunch of underscores.

Enjoy,
Charles.

Remove Office on Mac sux

2008-01-24T00:30:00.000-08:00

The Remove Office program that comes with the trial version of Microsoft Office just sucks. The two times I've used it, it's failed. The first time, I had my old iMac connected via Firewire (target disk mode) to my new iMac. I ran Remove Office on the new iMac to remove the trial version, and it removed the non-trial version on my old iMac - without even asking me which version to remove. Now, on my new MacBook, I tried running Remove Office (without any other computers connected), and it couldn't find the trial version to remove - never mind the fact that I was running Remove Office from the folder containing Office.

My advice to anyone is to just use 'rm -r' from the Terminal window. If you just drag it to the trash (without emptying the trash), the trail version will keep launching.

enjoy,
Charles.

Exporting from ExamView to Moodle

2008-01-20T16:12:00.000-08:00

I've just started using Moodle for a course (more on that later), and I wanted to import some questions I wrote using ExamView into Moodle. Although our version of Moodle (1.8?) suggests that it can import EvamView questions, it failed for me. I came across this post that suggests that it isn't even supported - at least not by the company that makes ExamView. Based on that post, here's what I ended up doing:

From ExamView Pro (version 4.0.8) export as Blackboard 5.x, which is a zip file.
Unzip the zip file.
Import the .dat file into Moodle as a Blackboard, not Blackboard V6+

(And, all of this is made more complicated by the fact that I've basically abandoned the PC for the Mac, and I only have a version of ExamView for the PC. Praise be to VMware for Fusion.)

There may be a simpler way to do this, but it works, which is all that matters right now. It would be swell if I could automate the process...

enjoy,
Charles.

Subersion on the Mac

2007-10-12T10:45:00.000-07:00

If you've ever used Subversion (SVN) on Windows, chances are you've used TortiseSVN. (If you haven't, definitely check it out.) It works as an extension to Windows Explorer. What this means, is that you don't have a separate UI for SVN - you do everything in Explorer. I was really struck by how cool this was when I was working on an old project using an old version of Perforce; it seemed barbaric to have to keep switching between the SCM UI and Explorer.

Until recently, there wasn't anything like that for the Mac, at least not anything that was free/open source and that I knew about. Then, along comes SCPlugin. From the web page: "The goal of the SCPlugin project is to integrate Subversion into the Mac OS X Finder. The inspiration for this project came from the TortoiseSVN project."

As of this writing, it's at version 0.7, which means it has a few rough spots, but for the light use I've put it to so far, it's been mostly pretty good. And since it's 0.7, it has nowhere to go but up.

enjoy,
Charles.

Seven Habits of Effective Text Editing - with Vim

2007-10-04T21:16:00.000-07:00

I'm a big fan of vim. It's the Unix vi editor with a large number of improvements. The biggest thing I like about it is that it runs pretty much everywhere - Unix/Linux, Mac, and Windows. Having a powerful, familiar editor on Windows was a huge technological leap forward for me when I spent a lot of time on Windows, and it's still very useful on those occasions when I'm "stuck" on Windows.

I recently stumbled across an old article (Nov 2002) from the creator of vim, Bram Moolenaar, that describes how to use vim more effectively. The big thing I learned from it is how to use ctrl-N to complete things like identifiers in programming languages - e.g., type "read" and it expands to "readlines" and offers you a list of other options in your program. Granted, this is old hat for fancy-ass IDEs like Eclipse and Netbeans, but I had no idea that little old vim had it. I guess that's what they mean by vim means "vi-improved."

enjoy,
Charles.

Very non-intuitive error message from SVN

2007-09-25T14:30:00.000-07:00

I just got the following error message out of Subversion (on the command line under Linux):

sh: ./svn-commit.tmp: Permission denied
svn: Commit failed (details follow):
svn: system(' svn-commit.tmp') returned 32256

It sounds like a file permission problem, but I couldn't find any problems - I was working in my own directory that I checked out of SVN. I ran strace to see what was going on, and right about the time that the Shinola hit the fan, I could see that it was trying to fire up an editor for the check-in comment. I looked, and I did not have the EDITOR environment variable set. Once I set it, it worked fine.

Let's see, "Permisssion denied" means that an environment variable was not set. Sure, that makes sense - NOT.

enjoy,
Charles.

Getting Maxtor OneTouch III to work on MacTel

2007-09-04T19:52:00.000-07:00

I recently bought a Maxtor (or is it Seagate?) OneTouch III external hard drive. It's a pretty sweet drive, maybe a bit big and noisy, but perfect for my needs - backing up my systems. One really nice thing is that it's totally Mac-centric: it ships preformatted with HFS+, it supports Firewire 400 and 800 (nice with the new iMac), and it includes Retrospect Express backup software, which originated on the Mac, as I recall.

It include a button on the front that you can program to do various things, the most obvious being to initiate a backup. But it didn't work; I'd push the button and nothing would happen. After poking around, I found this update for Intel Macs. After installing it (and rebooting), it works like a charm.

enjoy,
Charles.

What's in an OS X Package (.pkg) file?

2007-08-29T19:58:00.000-07:00

I recently had occasion to figure out how packages work on OS X. This isn't an extensive treatise on the subject, and I didn't bother to look for any docs. This is just some poking around I did in the Finder and mostly in Terminal.

The original motivation was that I wanted to uninstall a package that I'd installed. The package file is Subversion-1.4.4.pkg. The first thing I learned is that packages are another case where a directory (aka folder) shows up as a single file in the Finder. You can see the contents in the Finder by control-clicking on the package file and selecting Show Package Contents. The Finder opens up a new window (just like any other folder).

All I see in my package is a folder called Contents. Within that folder, is a file called Archive.pax.gz, which is a compressed pax archive. Pax is an archive program like tar and cpio. (Actually, it's an experiment on genetic engineering - in order to "solve" the tar vs. cpio wars, they merged the two and called it pax, Latin for peace.) The pax archive was subsequently compressed with gzip - hence the gz extension.

Too see what's in the archive, we need to decompress it and get pax to print a listing of the contents. I did this as follows, although there are other ways to skin this cat:

cd /tmp
gzip -d < Subversion-1.4.4.pkg/Contents/Archive.pax.gz | pax -v

(Pax contains an option to do decompression, but I'm too old fashion.) Note that I redirect the compressed archive into gzip. I did this because I wanted to leave the archive compressed. I could have decompressed the archive and then ran pax on that, but I wanted to leave the entire package unharmed. The first bit of the output looks like:

-rwxr-xr-x 1 root wheel 1664424 Jun 22 23:57 ./usr/local/bin/svnadmin
-rwxr-xr-x 1 root wheel 1571744 Jun 22 23:57 ./usr/local/bin/svndumpfilter
-rwxr-xr-x 1 root wheel 1664612 Jun 22 23:57 ./usr/local/bin/svnlook

So, we can see that these are the files that got installed (in /usr/local). One nice thing to note is that the path names are all relative - they begin with "./" rather than just "/". This means the files can be installed anywhere - change into a directory, extract the archive, and the files will show up in a directory called usr/local below your current directory.

How can I use all of this to uninstall these files? Again, there are many ways to skin the cat, but here's what I did. I extracted the archive to /tmp. ("I thought you wanted to remove the files. Why are you extracting them again?" Patience, my friend.)

cd /tmp
gzip -d < Subversion-1.4.4.pkg/Contents/Archive.pax.gz | pax -r

This creates all of the files, but under /tmp - e.g., /tmp/usr/local/bin/svnadmin. I can now use find to get me a list of just the files:

find usr -type f -print > /tmp/fff

I then looked through the file names in /tmp/fff, and they made sense, so I removed them all.

sudo rm -i `cat /tmp/fff`

The sudo command was needed because the files were not owned by my user ID. The package installer asked for the administrator password and installed the files owned by root (as I recall). Of course, I could have avoiding "installing" the files in /tmp by running the output of pax -v through some awk or perl, but the archive was small, and I knew the options to find off the top of my head - I would have had to look up some awk or perl, since I don't use them that often any more.

enjoy,
Charles.

P.S. This post almost never was: Blogger mangled the fonts repeatedly, and I almost gave up on it. Stupid JavaScript HTML editors!

Everything I Know About Business I Learned from My Mama

2007-08-22T19:47:00.000-07:00

Everything I Know About Business I Learned from My Mama by Tim Knox is nominally a business book. However, it is certainly not your typical business book. The author doesn't tell you what corporate structure (e.g., LLC vs. S-Corp) is best, how to keep the books, or how to manage a Fortune 500 company. Rather, this is a bigger picture view of going into business for yourself. Tim Knox is all about entrepreneurship.

Although this book isn't really a how-to manual with lots of nuts-and-bolts details. It would be a really good book for someone who is thinking of going into business for himself. It even begins with a bunch of reasons why someone shouldn't go into business. If after reading the book, someone still wanted to go into business, he should probably read a few more books before jumping in. (The E-Myth Revisited: Why Most Small Businesses Don't Work and What to Do About It is next on my list to read.)

The book is pretty short and humorous, and the chapters are nice little bite-sized chunks. Here's an attempt as his sense of humor: this would be a great book to keep next to the toilet - each chapter is about one visit long. (I have a special place in my heart for books like that.) I suspect the book was easy for Tim to write since he's been writing newspaper articles for some time. I can easily imagine that many of the chapters are extensions of articles he wrote for the paper, bless his heart.

One minor nit I could pick with the title is that there isn't all that much mention of stuff his mama told him. Rather, there's a fair amount of common sense that one's mother might impart.

I bought this book because I've been listening to Tim on Dan Miller's radio show - see www.48days.com. Dan is a career coach and wrote the book 48 Days to the Work You Love. The two of them spend a lot of time telling people to quit jobs they hate and move towards work they love. The radio show recently ended, and they've switched to an Internet format. The radio shows and the new Internet shows are available as podcasts.

enjoy,
Charles.

Roomba Rants

2007-08-22T16:27:00.000-07:00

I recently started using my (comparatively old) Roomba again, and I've been reminded of my chief complaint about Roomba: it gets dirty and needs to be cleaned.

Our house is a regular petting zoo, so stuff accumulates pretty quickly. This fills up the bin, which isn't so bad because emptying the bin is quick and easy. But the human hair and pet fur fowls the brush, and that takes some serious effort to clean. I suppose the Roomba for pets (which appears to be discontinued) addresses this with a brush that is comparatively easy to clean.

And then last night I discovered a new place to clean: the side sweeper brush. That thing was totally fowled (not surprising considering I just discovered that I needed to clean it - "read all the words"), and it took me 10 to 15 minutes with my pocket knife and tweezers.

I find it ironic that a cleaning tool requires a non-trivial amount of cleaning. Perhaps if I kept up with regular vacuuming, Roomba wouldn't get so overwhelmed by crap.

Finally, the reason I'm just getting back to using Roomba is that I just got a new battery after killing the old one some time ago. It turns out managing batteries with Roomba is a bit different than other devices I've used. Typically, I run devices down and recharge them. If I don't have an immediate use, I'd leave the device uncharged. Well, Roomba keeps discharging the battery even if it's not in use. And if you do that to a battery that's already low, it deeply discharges the battery - definitely not A Good Thing. I noticed the discharging thing: sometimes when I'd charge the battery and not use Roomba immediately, the battery would be low when I went to use it. So, Roomba prefers being left on the charger, even if you're not using it, which is something I tend to avoid with other devices. Anyway, the result was that I only got ~50 cycles out of the original battery - not good. The recording on Roomba's tech support tells you if you're not using the unit anytime soon, you should take the battery out. Fortunately, it's easy to install and remove the battery - e.g., no screws required.

Don't get me wrong: Roomba is still fun to use, and I did just put down more money for a new battery. So, I will continue to use it. It's just not as carefree and easy as I'd like to see it.

Later,
Charles.

New iMac

2007-08-17T21:46:00.000-07:00

Even though I was wishing for a Core2 Mini, when it was (finally) released (somewhat silently), I opted instead for one of the new iMacs, and I'm loving it.

My original plan was to get a Mini and use a KVM along with the PC boat-anchor that I keep around for a client. However, after using a KVM for a while with other PCs, I realized that KVMs can be kinda hokey. For example, my KVM makes a keyboard look generic - i.e., it hides any special keys. Also, although I have a cool 22" Viewsonic LCD, it's not as nice as an Apple display.

I opted for the 20" 2.4Ghz model with the stock 1GB of memory. I was very pleased to see how easy it will be to upgrade the memory when I get a few extra dollars. The 24" was pretty tempting (the price is quite reasonable), but I figured if I ever need a larger display, I can use the external video connected to my (somewhat inferior) Viewsonic for a ~40" display experience.

The CPU is hella fast, although I admit I'm comparing it to an 800 Mhz G4 iMac which was significantly taxes (~20% CPU) running Firefox and Gmail. To date, the only time I'm maxed out the two cores was converting audio files into MP3 with iTunes. My Internet is still only 128K ISDN, which makes for slow Gmail, but Steve Jobs can't be expected to fix that.

The new, thin keyboard is something that cannot be appreciated in a store standing over it. You have to sit in front of it with a real chair in a real working position to appreciate the nice ergonomics. My one gripe is that my thumb drive is too fat to fit in the USB sockets.

And finally, the Apple educational discount and promotions were sweet. After the rebates, I'll have a free Nano and printer, both of which I gave to the wife - score!

Anyway, Joe Bob says, 5 stars (on a scale of 4) - check it out!

enjoy,
Charles

Google Docs for Family Support

2007-06-13T16:19:00.000-07:00

I talked before about using Google Docs as a small-time wiki, but I've recently come up with another, slightly less geeky use for Google Docs. My sister and I live in Oregon. Our mother lives down in Orange County, CA. She was recently diganosed with stage 4 lung cancer and was temporarily hospitalized and then released.

My sister and I have been in the difficult position of trying to coordinate care from far. We both work full-time, and so we tag-team on calling people. As we call various professionals and friends, we learn new names and phone numbers of more people to call. (Sounds like Amway, but it isn't. :-)

Anyway, I set up a Google spreadsheet with the names and numbers I was working from and shared it with my sister. She added the information she had collected. We also have columns for email addresses (where relevant) and notes about the person (e.g., a friend from church, the social worker from the hospice, etc.).

The collaborative feature of Google Docs (and spreadsheets) is a life saver. And, if you're away from the Internet, just print it out and take it with you - how 20th century. Joe Bob says, four stars - check it out!

enjoy,
Charles.

Book: Windows Forensics and Incident Recovery

2007-03-01T19:54:00.000-08:00

Windows Forensics and Incident Recovery
by Harlan Carvey -ISBN 0-321-20098-5

This is a great book because I learned more than I thought I would from it. Coming from a command-line Unix background, I tend to view Windows as excessively GUI-centric (maybe that's why it's called Windows?) and full of opaque Microsoft voodoo. This books showed me that there are plenty of things to be learned from the Windows command line, and there are lots of transparent, open-source tools to expose the inner workings of Windows.

There are really three types of information in this book: how Windows works, tools to collect information about Windows, and the bigger task of forensic information extraction and processing. There is a lot of information about basic operating systems concepts (files, processes, etc.) and how they are implemented in Windows. I especially liked the presentation of user privileges - we typically only hear about those in the context of administrator versus non-administrator, but there is a listing of each of the individual privileges and what they mean. The tools that the author presents are primarily command-line tools, and many of them are written in Perl - very approachable for an old Unix hack. (A second edition of this book would benefit from a treatment on Microsoft's WMIC tool.) With the basic groundwork laid, the author presents a bigger picture of how to use all of the tools in a forensic investigation. He presents a series of dreams, which are a bit corny, but they serve as a sequence of case studies. He also provides a "forensic server" to storing all the little bits of information that get collected - a bit like "real" tools like EnCase.

Like the book File System Forensic Analysis, one of my favorite aspects of this book is that it provides a lot of practical information about applied operating systems - Windows. The author provides links to a lot of tools and web pages, so this book serves as an excellent starting point to learn a lot more about Windows and forensic data recovery. The text includes complete source code for the Perl tools, so a code-oriented reader can really see what the information is and where it comes from.

If I had to criticize something in this book, I'd say that Chapter 9 on scanners and sniffers drifts a bit from the central theme of the book, but then I've found that to be pretty common in security books because so many of the topics are interrelated; you start pulling one thread on the sweater, and the next thing you know, you've unraveled the whole thing.
All in all, this is a great starting point for learning about forensic data acquisition on the Windows platform.

Enjoy,
Charles.

Google Docs - Quick and Dirty Wiki

2007-01-09T21:39:00.000-08:00

I've been using wikis for collaborative software development for at least five years. I've typically used TWiki. It's a pretty cool tool, but the set-up is non-trivial, especially for a small project. Also, the last wiki I set up got spammed big time - it was on the Internet rather than a private intranet.
Google recently bought out JotSpot before I discovered it. It sounds like a cool idea - they set it up for you, and I guess it could be configured to be totally private, which would eliminate the spam. I'll be very curious to see what it looks like when it comes back online.
In the meantime, I've discovered Google Docs and Spreadsheets as a collaborative tool. At first I thought, free Word and Excel - who cares? I've already paid The Evil Empire for my software. But when you add the Internet storage and collaboration features, it becomes very cool. A shared document becomes a wiki page!
I see two really nice features with Google documents compared to run-of-the-mill wikis: the formatting and editing is a word processor (implemented in Ajax), not another markup language, and the sharing is on a document-by-document, user-by-user basis. In a small team, it's nice to allow Bob to see the document and Jane to edit it. On another document, they can both edit it. Of course, in a large team, configuring this one-by-one would suck.
Another nice feature is that you can upload documents (from Word, OpenOffice, others), so someone can begin something with a word processor (maybe when s/he is offline) and convert it trivially into a pseudo-wiki page. Sure you can cut and past from Word into a blog or wiki, but I hate the way some characters get mangled in the process.
The Google spreadsheets are nice for collaborative project management. I've been using Voo2do for managing simple task lists. It's nice, but the collaboration options are limited: you can share a password-protected, read-only view, but to allow someone else to edit, as near as I can tell, you have to grant them access to your whole account - not just the one task list you wanted to share.
With Google spreadsheets, all you have to do is create a spreadsheet with the tasks (ala Joel on Software), and share it, either read-only or read-write.
I'm currently working on a small project with two other people, and I've just gotten into this. So far, it's great. More news when it happens.

Enjoy,
Charles.

Core2 Mac Mini

2006-11-11T20:52:00.000-08:00

There has been a fair amount of speculation about when/if Apple will release a version of the Mac Mini using the Core 2 Duo processor. For example:

Apple hints at Core 2 Duo Mac mini?

How Soon Will the Mac mini Go Core2?

Personally, I'm dying to see a Core2 Mini. At the moment, I'm stuck with a 800 Mhz G4 iMac that just doesn't cut it any more, but I can't afford a 20" Core2 Duo iMac.

For the sake of argument, why would Apple not update the Mini to the Core2? Part of that depends where Apple really sees the Mini. When the (G4) Mini first came out, one of the pitches was for PC developers to use it in addition to a PC via a KVM switch. In this scenario, it made sense for Apple to make it beefy.

When the first Intel Macs came out, the Mini had a Core processor (I'll call the Core processor the "Core1" just to be extra clear) just like the other Macs, in particular the iMac. They wanted the Core archiecture, but there weren't many of those processors, so the Mini and the iMac were pretty similar. Then the Core2 came out and Apple bumped the iMac to a Core2 but left the Mini at a Core1.

I take this to indicate that Apple is trying to create some diversity in the product line - i.e., the need to differentiate the Mini from the iMac. Thus, they need to keep the Mini crippled, and they're no longer pitching the Mini as a developer machine. This is (unfortunately) like the IBM PC Jr back in the day. Now, they have bumped both Mini models to a Core1 Duo, but it's still a Core1 not a Core2 - i.e., still crippled.

The other possible motive (for Apple) of keeping the Mini at Core1 is profit margin. When the Core2 came out, Intel slashed the price of the Core1, but Apple has not dropped the price of the Mini. If Apple had a big stock of Core1 processors (especially Core1 Duos) when the Core2 came out (not likely given how shrewd Apple often is), this gives them a way to flush their Core1 inventory. More likely, Apple is just making bank on the reduced cost of the inputs.

So, I can see a few reasons why Apple may not bump the Mini to a Core2 very soon. I hope I'm wrong it because I'd really like a Core2 Mini. The MacWorld release timeframe that is being rumored would fit my budget very nicely.

enjoy,
Charles.

Flex: Debug Flash Player on OS X

2006-08-24T18:07:00.000-07:00

As noted in "Building and Deploying Flex2 Applications" (Live Docs), the output of trace statements can be configured in the standalone debug version of the Flash player. The documentation describes the location of the mm.cfg file on Mac OS X using Mac-style (non-Unix) path notation with colons to separate directories.

I set up a config file, but for the life of me, I couldn't find the output file. Then, I found an old, pre-Flex2 blog post by Christian Cantrell which fills in two missing parts of the documentation. First of all, if no TraceOutputFileName directive is given, the trace output will go to flashlog.txt in the same directory as the config file - fair enough. Once I removed the directive, I found the output file.

Also, if a path name is given, the path must be specified in the old Mac-style notation (with colons) rather than Unix-style (with slashes). I suppose that's why they specified the location of the config file in that notation, but the significance of the colons was lost on me. One totally trivial tidbit correctly noted in Christian's blog post is that the directory name is "Macromedia" not "macromedia", as indicated by the docs. (Since the Mac's HFS+ file system is basically case insensitive, it doesn't really matter.)

Once I figured that stuff out, I could get my trace output and get on to fixing my bugs.

I do have (at least) one gripe with the debug Flash player on the Mac compared to the PC - it doesn't keep a history of recent URLs. So, every time you go to open an URL, you have to type it in again (or paste it from a browser, which does keep history.) Even if you keep the player running, it still doesn't remember the last URL it opened.

Enjoy,
Charles.

Flex: Sending Messages from Server-Side Java Code

2006-07-19T19:56:00.000-07:00

I've been trying to figure out how to send messages from the depths of my Java application out to Flex clients. I'm not talking about messages that originate on a client, pass through the Java server, and then head back to the clients - that stuff is documented in the manual. I mean there is a process going on in the server that needs to send messages to the clients.

Of course, JMS and the JMS adapter would work, but I've been really leery of the complexity of JMS since it came out years ago. It has the advantage that it would allow my application to be separated from the Flex server, which would work well in terms of scaling. I may yet try JMS, but for the moment, I'm looking to avoid JMS.

So, I was thinking that writing a message service adapter would be the key. Adobe has a brief example of creating a custom Message Service adapter. It seems to be a complete example, but it left me with lots of questions. For example Message is an interface, which concrete class do we use? How do we construct it? What all can we do with the MessageService? What value are we returning from invoke?

After some digging around in the sample code, I came across a couple of Java classes associated with the dashboard sample application. So, I've taken the code from that example, paired it down a bit, and wrote a very simple, Soviet-style (butt-ass ugly but functional) Flex interface for it.

Below is the server-side Java code.
import java.util.*; import flex.messaging.MessageBroker; import flex.messaging.messages.AsyncMessage; import flex.messaging.util.UUIDUtils; public class FmsDateFeed { private static FeedThread thread; public FmsDateFeed() { System.out.println("FmsDateFeed.ctor: " + this); } public static void main(String[] args) { FmsDateFeed feed = new FmsDateFeed(); feed.toggle(); } public void toggle() { if (thread == null) { System.out.println("FmsDateFeed.toggle: starting..."); thread = new FeedThread(); thread.start(); } else { System.out.println("FmsDateFeed.toggle: stopping..."); thread.running = false; thread = null; } } public static class FeedThread extends Thread { public boolean running = true; public void run() { MessageBroker msgBroker = MessageBroker.getMessageBroker(null); String clientID = UUIDUtils.createUUID(false); System.out.println("FmsDateFeed.run: msgBroker=" + msgBroker + ", clientID=" + clientID); Random random = new Random(); while (running) { Date now = new Date(); AsyncMessage msg = new AsyncMessage(); msg.setDestination("time_msgs"); msg.setClientId(clientID); msg.setMessageId(UUIDUtils.createUUID(false)); msg.setTimestamp(System.currentTimeMillis()); HashMap body = new HashMap(); body.put("userId", "daemon"); body.put("msg", now.toString()); msg.setBody(body); System.out.println("sending message: " + body); msgBroker.routeMessageToService(msg, null); try { long wait = random.nextInt(8000) + 3000; System.out.println("sleeping: " + wait); Thread.sleep(wait); } catch (InterruptedException e) { } } } } }
The toggle method is called from the client to turn the message stream on or off. When messages are turned on, a thread is created and the run method is called. It just loops creating Java Date objects and putting the String representation of the date into Flex messages - instances of AsynchMessage. The key for me was the routeMessageToService method on the MessageBroker class. When you have the class compiled, install the Java class file in samples/WEB_INF/classes.

The client is pretty simple, by comparison. It has two features: a button for turning the message stream on and off (via a RemoteObject call) and an event handler to process the incoming messages.
<?xml version="1.0" encoding="utf-8"?> <mx:Application xmlns:mx="http://www.adobe.com/2006/mxml" backgroundColor="#FFFFFF" creationComplete="initComp()"> <mx:Script> public function initComp():void { consumer.subscribe(); } public function toggle():void { feeder.toggle(); } import mx.rpc.events.ResultEvent; import mx.messaging.events.MessageEvent; public function messageHandler(event:MessageEvent):void { var body:Object = event.message.body; output.text += body.userId + ": " + body.msg + "\n"; } </mx:Script> <mx:RemoteObject id="feeder" destination="time_msgs"> <mx:method name="toggle"/> </mx:RemoteObject> <mx:Consumer id="consumer" destination="time_msgs" message="messageHandler(event)"/> <mx:Form> <mx:FormItem> <mx:Button label="Toggle Date Feed" click="toggle()"/> </mx:FormItem> <mx:FormItem label="Output Message"> <mx:TextArea id="output" width="250" height="150" /> </mx:FormItem> </mx:Form> </mx:Application>
The interesting features are the configuration information for the RemoteObject and the message Consumer. These correspond to destinations configured on the server (see below). When the toggle button is pressed, we just call the toggle ActionScript method, which make a call on our RemoteObject, feeder; we don't care about a return value, which simplifies things.

We configure the Consumer to call our messageHandler method when new messages arrive. It just gets the message body and appends it to our text box. Install this file in somewhere in the samples directory - I created my own subdirectory for the dumb, little programs like this.

To configure the server, edit messaging-config.xml in samples/WEB-INF/flex and clone the destination for "dashboard_chat" and call the new version "time_msgs". You could change the Java and MXML to use the existing dashboard_chat destination (which is what I did initially), but a) you could end up with a conflict with the Flex demos, and b) writing to messaging-config.xml (or any of those other config files) is a convenient way to get the server to restart and see your new Java class.

Oh yeah, you also need to edit remoting-config.xml to contain a RemoteObject destination also called "time_msgs". (You could use separate names for the two destinations, and it would probably be less confusing, but I was too unimaginative.)

If all goes well, you should get messages with the current time showing up randomly in the Flash client.

Enjoy,
Charles.

Flex: The Second Simplest Example of RemoteObject

2006-07-12T17:56:00.000-07:00

(Update: I wrote this some time ago, and I don't have access to the source code any more. Therefore, I can't send it to you. And, I don't know about newer versions of Flex.)

In a previous post, we developed what I claim to be the world's simplest example of using the RemoteObject facility in Flex 2 to invoke a method on a Java object using the RPC services. In that example, we just had a button on a Flash application that called a method on the server. All the method did was print a message. Although we passed a message into the server method, that message was hard-coded in the ActionScript, and no results were passed back.

In this post, we will build on that to pass some user data into the server and get a response back. In theory, this ought to be very simple: the user data just comes from an input box, and the result will just be displayed in a text box.

Java Code

Below is a simple function that you can add to the SimpleRemoteObject from the previous example:
public String echoMessage(String msg) { System.out.println("SimpleRemoteObject.echo: " + msg); return "The RemoteObject says: " + msg; }
As you can see, the method writes the message to the console (like before), and returns the message with a little text prepended to it. Pretty simple.

Flash Client

The user interface is pretty simple, and I just added it to the form in the previous example. We have a TextInput to enter data, a Button to invoke the method, and TextArea to display the result from the server.
<mx:FormItem label="Input Message"> <mx:TextInput id="input"/> </mx:FormItem> <mx:FormItem> <mx:Button label="Echo Message" click="echoMessage()"/> </mx:FormItem> <mx:FormItem label="Output Message"> <mx:TextArea id="output"/> </mx:FormItem>
So far, so good. The button calls an ActionScript method (echoMessage) to call the server - just like the previous example:
public function echoMessage():void { remoteObj.echoMessage(input.text); }

We just get the text from the TextInput object (input) and call the method on the server. (Note that although I named the ActionScript method the same as the Java method - echoMessage - this isn't necessary; I'm just not creative enough to think of anything else.) But wait, something is missing: what about the message coming back from the server? Based on my experience with Java (and dozens of other languages), I would expect to see code that looks like:
output.text = remoteObj.echoMessage(input.text);
But, that's not how it works in Flash. Flash makes RPC calls asynchronously. This means that Flash doesn't wait for the result to come back from the server, and that's actually "a good thing." Without asynchronous server calls, the Flash client would hang during a (potentially long) server call, and that would be a bad thing.

OK, so how do we get the result back from the server? This introduces some additional complexity, but it's still tractable. It begins in the MXML to describe the server method:
<mx:method name="echoMessage" result="displayResponse(event)"/>
This says that when the echoMessage method on the Java server object is called, and the result arrives at some later time, the displayResponse ActionScript method will be called.

This method is pretty simple, too:
import mx.rpc.events.ResultEvent; public function displayResponse(event:ResultEvent):void { output.data = event.result; }
That's all there is. Note that we don't even have to update the remoting-config.xml file on the server. This is because we are not creating a new destination; we're just using the same Java class - SimpleRemoteObject.

When you run the application in the browser, enter a message in the input area, click the button, and the response will eventually appear in the output area.

If you need the complete code, let me know - post a comment. At the moment, I'm too busy to put together a zip file and post it somewhere.

enjoy,
Charles.

Flex: The World's Simplest Example of RemoteObject

2006-07-12T17:00:00.000-07:00

Here is a very, very simple example of using the RemoteObject RPC services in Flex 2 to invoke methods on a Java object. I came up with this because I was struggling to figure out all the stuff needed to do this. In theory, it's pretty simple, but there are a number of components to deal with, and they all get put in different places on the server, and a bunch of those places look very similar to other places on the server.

Introduction

It was especially difficult for me because I'm new to Flex, and I'm coming at it from a server-side background in Java. I don't care about making pretty user interfaces (that's someone else's job) - I'm the guy who has to make the server code work. However, most of the documentation assumes that you're coming at it from the other end - the client side.

I couldn't find any simple examples of using RPC in Flex 2.0. I foolishly started with Flex 1.5 example and wasted a bunch of time updating the syntax in it in order to get it to compile.

After I got that to compile (from the command line - I'm old school), it wouldn't run. After I got the Flash debugger working, I could see that the messaging destination didn't exist, but I couldn't figure out why. So far as I could tell, I had configured it correctly, but there was no indication that things were making it to the server. Finally, I gave up my command line and put the MXML file on the server, and just let the web-tier compiler do it for me.

And, it just worked! The documentation makes reference to using the SDK compiler versus the web-tier compiler, but it just didn't click with me. But, it was still more complicated than I needed. So, I created the following, to prove that I could do it, and that I knew all the voodoo needed.

I will assume that you have the Flex Data Services demo server with Jrun installed an running. I will specify all paths from the root of the jrun server.

Java Code

Let's start with the simplest function in a Java file - basically, "hello world."

public class SimpleRemoteObject { public void writeToConsole(String msg) { System.out.println("SimpleRemoteObject.write: " + msg); } }

This will just write a message to the server console window. (We don't even have to use any fancy Java logging.)

Compile that file and put the SimpleRemoteObject.class file in the directory jrun4/servers/default/samples/WEB-INF/classes on the server. (Note that since the class is in the unnamed, default package, we put the class file in classes, not a subdirectory.)

Flash Client

Now, we need a simple Flash app. Here's a simple MXML file:
<?xml version="1.0" encoding="utf-8"?> <mx:Application xmlns:mx="http://www.adobe.com/2006/mxml" backgroundColor="#FFFFFF" initialize="initApp()"> <mx:Script> public function initApp():void { } public function writeToConsole():void { remoteObj.writeToConsole("hello from Flash client"); } </mx:Script> <mx:RemoteObject id="remoteObj" destination="sro"> <mx:method name="writeToConsole"/> </mx:RemoteObject> <mx:Form> <mx:FormItem> <mx:Button label="Write To Server Console" click="writeToConsole()"/> </mx:FormItem> </mx:Form> </mx:Application>
Create a directory called sro under jrun4/servers/default/samples and put the MXML file there.

Now, we can compile the file by loading it in the browser with a URL like:
http://192.168.123.130:8700/samples/sro/SimpleRemoteObject.mxml
(The key is that the file is under /samples/sro based on where we copied the file.)

Don't click the button yet! We still have to configure the FMS destination. Just make sure it compiles correctly.

Configure the Server

Add the following XML code to jrun4/servers/default/samples/WEB-INF/flex/remoting-config.xml. Just put it within the service tag for the remoting-service - e.g., near one of the other FMS destinations.
<destination id="sro"> <properties> <source>SimpleRemoteObject</source> </properties> </destination>
And now, if you press the "Write to Server Console" button, the message, "hello from Flash client" will appear in the Jrun server window.

That's it. It really is as easy as Adobe says it is...
once you figure it all out.

enjoy,
Charles.

Tip: Managing Applications on OS X

2006-07-07T11:19:00.000-07:00

I've heard that some people complain that Mac OS X doesn't have anything comparable to the "Start" button on Windows. I've never felt that way, but I have to admit, I'm amazed that some people always use the Finder to navigate to the Applications folder to launch application. If that's your way of starting applications, then I can see why you'd think you need a start button.

Below are some really simple tips to manage the applications and get the "Start button" functionality using the existing Dock.

Put Applications in the Dock: Begin by dragging the Applications folder to the dock. Open a Finder window, grab the Applications folder, drag it down to the dock, and let go of it to the left of the vertical bar that separates the applications from the Trash Can. Now, you have a "Start button." To use it, bring the mouse down to the dock, position it over the Applications folder, click and hold (for a second or two), and the list of applications will shoot up from the folder. Move the mouse over the application you want to launch and (finally) let go of the mouse button.
I've used this from the very beginning of when I started using OS X. It's so simple, but it makes the system so much easier to use. It drives me up the wall to work on a stock OS X system where you have to use the Finder to go to Applications to launch an application.
Create local applications folder: Once you've installed more than a few applications on your Mac, the "start button" we created above will be too big - you'll have to scroll to find the application you're looking for. What you need is to organize the applications in the Applications folder by creating some folders within the Applications folder. Therefore, create a folder for your new applications. I called my folder "Local Applications." Anytime you install new applications, put them in there instead of Application.
If you have a bunch of existing applications, it seems to be OK to just move them into your folder. However, take it from me: do not move any of the Apple applications - when Software Update runs, it will install things in the original location. Now, those applications are available from the "start button" under the name for your folder - e.g., "Local Applications."
Create a short list of favorite applications: Even with the benefits of the organization we set up above, the list is still long, and it contains a bunch of applications that you hardly, if ever, use. Create a new folder pretty much anywhere - e.g., in your home directory. Open a separate window in the Finder and navigate over to Applications. For any applications that you use regularly (either in Applications or Local Applications) create an alias of the application (right click/control click on the application icon and choose "Make alias") and drag the new alias into your new application folder. I also include aliases to Applications, Utilities, and Local Applications in my folder.
Finally, drag your new folder into the Dock (and optionally remove the original Applications folder you put there). From this new "start button," you can quickly reach the applications you use regularly, and if you included Applications in your folder, you can reach everything else, too - the best of both worlds: quick access and complete access.
Change the icon: Optionally you can change the icon of your applications folder - I copied the icon from the original Applications folder into my personal applications folder, but you can use almost icon.

enjoy,
Charles.

Book: File System Forensic Analysis

2006-06-26T20:56:00.000-07:00

File System Forensics by Brian Carrier. ISBN: 0321268172.

To my way of thinking, this is a really good book about file systems, that just happens to use forensics as a unifying theme and framework under which to study the file systems. The book provides the most detailed coverage of file systems I've seen, short of reading the source code. I used it as a textbook in an advanced operating systems class, but it is not really a textbook, per se.

The author begins with an introduction to the concepts behind digital forensic investigations. He continues with a ground-up introduction to disk drive technology and how disks are used in computer systems. The introductory material concludes with a generic framework for discussing the components and characteristics of file systems.

With all the groundwork laid, the meat of the book consists of detailed discussions of FAT, NTFS, Ext, and UFS file systems. Each file system is presented at a high level first, followed by a detailed description of the structures on disk. The high level information is presented with pictures and via output from the author's file system toolkit (The Sleuth Kit). The details are presented with tables of structure members without resorting to C code, which makes it easier to see the trees rather than the forest, especially for non-programmers.

I found the information about the Microsoft file systems (FAT and NTFS) especially useful, since there isn't much real documentation on those file systems available, and a lot of what is available seems like rumors spread at recess in a schoolyard.

In conclusion, this is a really good, perhaps even the best, book on file systems, even if you're not into forensics. If you're looking for serious details about file systems or forensic analysis of file systems, this is your book.

Enjoy,
Charles.

Welcome to Western Skies

2006-06-14T12:29:00.000-07:00

Welcome to the Western Skies technology blog. The purpose is to share information related to the technologies that I am involved with. Eventually, this will grow into the corporate web site for the Western Skies Technology Group with a real domain name, but that will require paying off the domain squatters that have the domains I'm interested in.