FloorSweepings

What am I looking for?

2011-09-21T19:39:00.000-07:00

20110921: I'm looking for work. Here's what I'm thinking would be ideal (this changes because I learn something new on every project - I'll keep this post up to date):

A long term (contract or perm) engagement with a team working on legacy apps. That's where the real challenge is - greenfield is easy. I'm dead serious about this. I know: every developer wants to get in on the ground floor of the next big cloud/social/multiplatform/HTML5/your-hot-language-here killer app. Not me. I do that at home. At work I want a challenge of a particular type: the combination of technology and social interaction that is Agile and the jungle of legacy code.

Notice I say "team". Ideally this would be a functioning XP team (if you think I'm talking about an old but still kicking OS, our relationship is already in trouble). Or - better: a newly formed team in an organization that sincerely wants to go Agile and understands that Scrum is about management, not development: XP is a Agile methodology for developers.

I recently had a disastrous stint with a company that did not understand Agile. I was actually told not to talk to other developers about what I was working on, because they were "busy". Right. These folks misrepresented themselves as open to Agile. They weren't - or more charitably, maybe they just didn't know. Oh sure, they had daily standups. But in terms of development process they were mostly in the Cowboy/Hero mode.

As a matter of fact, I'm a terrific team member and mentor. I always work Test-Driven (TDD and BDD) even if nobody else is, but I'm glad to show teammates how productive it can be - I gained some converts on the project I'm now rolling off. (If you have a problem with TDD/BDD - again, our relationship is already in trouble.)

I am absolutely not interested in being an architect, QA tester, business analyst, DBA, Websphere administrator, etc., etc. I'm a software developer: J2EE, DotNET or Rails, frontend, middle or backend - ready willing to learn new OO or functional languages and frameworks and work in new domains. (Of course, as an XP developer I have done all of the jobs above - as part of my normal development duties.)

Yes - I need to put food on the table and in bad times I take bad jobs. Fortunately this is not a bad time. But you'll have to be honest with me: if you think Agile - XP in particular - is a bad idea, or it's being forced down your throat by Corporate and you'll show them!, better you don't even phonescreen me.

20110922: Did I mention that I live and work in Chicago and will not travel or relocate?

20111011: Kent Beck recently said a lot of what I'm thinking in My Ideal Job Description.

Testing and Design

2010-07-13T10:58:00.000-07:00

Just watched a great video - The Deep Synergy Between Testability and Good Design, by Michael Feathers. Combining this with JBrains' blog post How test-driven development works (and more) and his video Integration Tests are a Scam would make an excellent intro to TDD, and maybe I'll get a chance to use it soon.

Not so fast, Otaku

2010-07-07T09:00:00.000-07:00

That last post was a little over the top, particularly the part about "good, true and beautiful".

Strike "good".

Although XP does have an ethical component in terms of how team members should relate to each other (and, to some extent, to the customer), it doesn't deal with some wider issues.

You might never do a website for Kim Jong Il or Osama bin Laden. That's an easy call.

What about Goldman Sachs or BP?

That's me at the pairing station, practicing my religion

2010-07-06T22:43:00.000-07:00

The metaphor of programmer-as-craftsman and the concept of apprenticeship are great up to a point ... and that point is where it gets medieval.

I don't think software development is a craft - I think it can and should be (and - in XP teams - is) an alloy of art, science and religion.

For models, I look first to the Renaissance, to people like Michelangelo, who studied anatomy as seriously as any practitioners of "scientific medicine" of the time, and to DaVinci, who studied physics equally seriously. They were both also brilliant artists.

The medieval role that comes closest to a model for me is not the craftsman but the monk - particularly the scribe whose faith in a divine perfection drives the work.

But the scribe is a soloist. And so were Michelangelo and DaVinci.

Renaissance artists who weren't simply soloists were businessmen. The master/journeyman/apprentice hierarchy in the art biz survives even now - e.g., Andy Warhol and his "factory".

What happens when software development organizations use the guild as a model? In two big "Agile" consultancies I know of and will not name here, the result is ... architects. Oh yes, they code. But they also do BDUF. And they radiate "authority". This doesn't meet my understanding of Agile.

XP is not a solo activity but a communal religious action with a particularly clear vision of the sacred. In OO we have an esthetically and logically balanced basis for design, still evolving. In XP, we have a concept of perfection - the sacred - that is dynamic and minimalist: the simplest system that does exactly what the customer wants, no more no less. (I'm using the term "sacred" here the way social anthropologists since Durkheim have used it - there's nothing intrinsically supernatural about it, except to the extent that symbols themselves are supernatural [for which I will make a case elsewhere].)

The closest modern analogy might be a Charismatic congregation (Catholic, Protestant or ...?) with a lay leader who is not a priest (architect), who has no special connection to the sacred. These groups are radically democratic.

What makes XP more than any of these models is the unification of art, religion, science and commerce.

It's the wisdom of a small crowd focused on the good, the true and the beautiful.

Nano-nit

2010-07-06T08:35:00.001-07:00

As I've said before, JBrains is a freakin' genius. I've never disagreed with anything he's said - except now for this one trivial point: about 27 minutes into "Stop It Or I will Bury You Alive In A Box"* at Agile Bengaluru 2010, he says that unit tests without assertions test nothing.

Well, it's not quite nothing: if the test actually executes some target code and doesn't throw an exception, it's testing something. But it's nano-next-to-nothing, it's still crap, and JB is still right.

* Okay, two nano-nits: this is not correct title case.

------------------

P.S. I forgot to say that you have to watch this whole presentation. JB is - as usual - moving the state of our art-science forward.

More hardlining

2010-06-16T11:47:00.000-07:00

Matt Heusser says

If your company is just starting to talk about "agiling up" it's software process, my advice is to forget about timeboxed iterations and test driven development. Oh yes, those are good things, and they can enable agile software development, but let's be very clear: if the team isn't self-directed, cross-functional and respecting of its people, it's not Agile software development. It's ... something else.

Hold on to those ideals. Communicate about them clearly. Fight for them, leave no room for misunderstanding, and you might just make your little part of the software development world a better place.
I think that is noble, and worth fighting for.

Read the whole thing by Matt Heusser.

Hardlining

2010-06-14T07:38:00.000-07:00

I wrote the following toward the end of my last contract. I hesitated to post it because I didn't want to offend the folks I worked with, and because I worried about the effect it might have on my own marketing. With some distance now, it doesn't seem that offensive, and if it eliminates some potential opportunities, they're likely to be in places I wouldn't want to spend 40 hours a week in anyway.

My current "engagement" has turned me into a hard liner. I was burned, and I saw the light. I'm not going to go into details, because I don't want to be blacklisted as a troublemaker.

But I suspect I have no choice: I am a troublemaker.

I'm a true believer, and what I believe is this:

There is no Agile but XP and success is its profit.

Anything else that calls itself Agile is just a compromise with an increasingly myopic business/bureaucratic culture obsessed with short-term profit and the illusion of control and predictability via hierarchy.

Scrum may rein in some of the worst tendencies of that culture, but it doesn't get at the heart of the problem.

XP is about two things: one is communication - the dynamic perfection of process.

The other is perfect software.

This is not the static Baudelairean perfection of BDUF. It's dynamic, minimalist, infinitely flexible. To be an XP developer is to be a Zen calligrapher inscribing sacred language, to apply all one's intellect and energy to creating and maintaining software that does exactly what the stories say, no more, no less.

XP originated in the context of OO. Unlike relational databases, OO didn't spring full-blown from simple, elegant mathematical models, but it has evolved into something damn close.

This vocation is absolutely incompatible with the hierarchical/centralized model in which coders have no autonomy.

The bureaucratic model runs on the differential presence of one human emotion: guilt. You rise to the top of a bureaucracy by becoming (or being born) a sociopath, pushing risk downward. Those stuck at the bottom accept this burden because they are guilty. The vision of future reward, through graduation to heaven, winning the lottery, or the application of sheer pluck and grit, is a consolation, not a motivating force.

I've been there - recently - I know exactly how that guilt works. People who work in the environment I'm talking about would recognize the power of that guilt - if not for the protective psychological repressions and distractions - the bread and circuses - that make it bearable.

XP flourishes in startups and small businesses that can't afford IT departments.

But I want to take it into the belly of the beast. XP is strong medicine, tough love.

Part II - Heroism

Waitaminit! The success of TradDev [i.e. waterfall/hierarachy/cmd&ctl] is not all about fear. There's a carrot, not just a stick. It's the solitary pleasure of heroism. And the dark side of that is empire building and job-security-thru-opacity.

I had a heroism moment of my own: successfully refactored some legacy Flex code without a net - i.e., no tests. That was scary, and given the pathological coupling of the undocumented/untested codebase, it could have been a disaster. (Actually, it still could be. While I'm on that subject, instead of working effectively with legacy code by building a test harness around it, the team is drawing diagrams to document it. This will just contribute to its rigidity.)
...
As I write this, two heroes are talking near my desk. It's a happy conversation. They're reinforcing each other's sense of being on top of things, and the buzz of working without a net in a risky environment. So what if it's about BDUF?
...
Now I'm idle. No corkboard with cards for this iteration waiting to be picked up, no pairing. Waiting to get a JIRA (i.e., a task) assigned to me by our manager, who's working from home. Yet another example of waste in the TradDev world. I'm here waiting, so I'll bill for it, but it feels nasty. If I offered to go home, what would happen?

Agile Manifesto Reprise - This Time it's Personal

2010-05-21T13:00:00.000-07:00

Stole this from Ron Jeffries and I'm posting the whole thing cuz it's my story too except for maybe the cars:

> Where will YOU be in 3 years?

Well, in my case I want to be alive and in good mental and physical
health.

Looking back over various places I might have "wanted" to be, here's
what I think today:

I thought I wanted promotions, to be given more "power" as a
recognition of how good I was. This reflected the organization I was
in, which valued power. I was wrong. I didn't get a lot of joy from
power. I did get joy from being able to solve harder and larger
problems by having more people working on them with me.

I thought I wanted more money. This reflected my good taste in cars,
and has certainly given me some pleasure. However, the highest
paying jobs I have had were also the least pleasurable, and
ultimately my lack of pleasure in them caused me to perform poorly.

I thought I wanted more women. You can imagine how that worked out.

I thought I wanted to be like people I admired. Many of them were
major jerks (even more so than I am naturally) and emulating them
did not serve me well.

What I want, today, is to spend every possible moment doing things
that I enjoy, with people I enjoy doing them with. I enjoy doing
things that I do well, and things where I can sense that I am
improving. I enjoy doing things that other people appreciate.

Clarification of previous post

2010-05-17T09:45:00.000-07:00

Actually, that last post was a nitpick - things are going well on this project. My bottom line is in the black: the question I've learned to ask myself about a job is whether it is an energy source or an energy sink, and this one has become a source. The team is coalescing, including management, customer reps, etc., in spite of the legacy ceremony.

Trust/coherence is building, and the only question is whether/when management will perceive this and start dismantling the ceremonial edifice.

Efficiency and Trust

2010-05-16T20:33:00.000-07:00

Haven't posted for a while because I've started a new contract. The team is nominally Agile, the customer is nominally on board, but ... not really. There's a lot of ceremony, no pairing or TDD expected (although we would not be punished for doing it), a lot of micromanagement of time with JIRA (be sure to charge that 15-minute daily standup to the right task!).

And I finally made the connection....

What do you do when your team/company is all excited to be “agile”, while your customers want to operate in the traditional, heavy-weight, documentation-driven approach? ... In most cases it turns out that they don’t trust the software team and hence they want to push all the risk to the software development side.
-- I Say Agile, You Say Traditional, Document-Driven

Ceremony is the cost of doing business in an environment without trust. Agile is efficient because trust allows us to eliminate ceremonies of security/CYA/blame/risk-shifting, which waste resources.

The word "ceremony" as used in Agile circles could almost be glossed as "institutionalized Muda (waste)". As an anthropologist, I think of it from the perspective of Durkheim: He saw religion as a mechanism that shored up or protected a threatened social order.

For example, estimates "cast in stone" are sympathetic magic, telling a highly structured story in the hope of imposing certainty on a situation involving major unknown risks. Estimates in a traditional environment are not a tool but a prayer - or an oath.

...

I will not identify the customer or the consulting company I'm subcontracting through. Not for any reasons of trade secrets or confidentiality, but because of this:

The customer's dress code is the new biz casual - i.e., jeans, polo shirts and sneakers are fine for everyone. The consultancy requires us to wear the old biz casual - no jeans, no sneakers, dress shirts. I was told this is because we want to impress the customer by being "a level above them".

Consistent so far.

But then - we have casual Fridays! Not the customer - they're always casual. Just the "one-level up" consultants.

WTF?

It's like

... but because we really care about all our ... uh ... development associates ... , no beatings on Friday.

Moo-dada!

I won't believe a consulting company is Agile as long as it carries the baggage of the Old Biz Religion.

[UPDATE: see the clarification.]

Rockclimbing, Baguettes, Agile Skills and Opposing Forces

2010-04-15T21:19:00.000-07:00

Got into some interesting conversations recently - check out Slack: we're not slacking off! and FORCES Behind Skills.

Life/Death/Agility 6: How Many Deaths Will it Take?

2010-04-02T10:25:00.000-07:00

Milo passed this little gem on to me. It was published eight years ago and it makes just about every argument I've been making in this mini-series. Why so long and still no change?

The passage quoted below may explain why "critical XP" still isn't happening: a chicken-and-egg problem. Read the whole thing for details on "Ron's project" and "the audit".

The theory of punctuated equilibrium holds that biological species are not likely to change over a long period of time because mutations are usually swamped by the genes of the existing population. If a mutation occurs in an isolated spot away from the main population, it has a greater chance of surviving. This is like saying that it is easier for a strange new fish to grow large in a small pond. Similarly, disruptive technologies (new species of technologies) do not prosper in companies selling similar older technologies, nor are they initially aimed at the markets served by the older technologies. Disruptive technologies are strange little fish, so they only grow big in a small pond.

Ron’s project was being run under a military policy, even though it was a commercial project. If the company policy had segmented off a commercial area for software development and explicitly allowed the team to develop its own process in that segment, then the auditor would have been satisfied. He would not have seen a strange little fish swimming around in a big pond, looking different from all the other fish. Instead he would have seen a new little fish swimming in its own, ‘official’ small pond. There XP practices could have thrived and grown mature, at which time they might have invaded the larger pond of traditional practices.

But it was not to be. The project was canceled, a victim not of the audit but of the economy and a distant corporate merger. Today the thing managers remember about the project is that XP did not pass the audit. The little fish did not survive in the big pond.

...

[in case you don't recognize How many deaths will it take...]

Learning as a design skill

2010-03-29T19:37:00.000-07:00

the question isn't how good you can make it, it's how fast you can learn about it. that's a design skill too.

-- Kent Beck tweet

Can we come up with "learning cycles" as powerful and satisfying as Red-Green-Refactor?

Hmm - downshifting tweets to blog posts ... too retro? I would have retweeted it, but nobody follows me on Twitter. At least here there's a chance folks will check out the blog later....

Anything wrong with this picture? - 2

2010-03-29T19:07:00.000-07:00

"Our team was perfectly aware that we sold scareware," says a translator who worked for the company in Kiev in 2008. "The manager never made a big mystery of that." The team the translator was part of had 10 staff and 15 freelancers to translate the text of IM's products into 28 languages. "Not everyone was happy about it, but money is money," the translator says. IM was paying around 60 per cent more than similar jobs elsewhere offered.

A mid-level employee, who left three years ago after realising what the company was doing, says that initially IM employed skilled developers to create genuine products. As managers became increasingly concerned with making money, quality declined and the fake scans came into use.

Roughly half the people working there knew the full story, says the employee, but again money talked. "There were a lot of young people working there who did not care about the product. They just took their salaries."

-- 'Infections found': Inside the great scareware scam

Anything wrong with this picture?

2010-03-27T12:31:00.000-07:00

As an example, consider one of our first Agile projects. Its goal was to provide field representatives with handheld clients to access a back-end application for submitting and tracking loan applications for high-end recreational purchases such as boats, airplanes, and exotic cars. Without such a tool, the field representatives could not tell customers whether their loan applications had been approved for several days. In that time, customers often came to their senses and canceled the purchase. To finalize more sales, we wanted to be able to approve loans and notify customers while they were still in the showroom and still excited about the purchase. Furthermore, the business unit manager had calculated an opportunity cost for delayed delivery of the solution: If we missed the spring sales rush, we could lose up to $2 million in revenue per year, as well as losing market share and conceding the market leadership position to a competitor. That is an example of an urgent project. (The traditional IT group put in a bid of 10 months at $850,000; we did it with 4 people in 6 weeks at a cost of $73,000.)

-- from the otherwise great Dave Nicolette post that deFUDs Boehm and Turner (cited earlier in the Life/Death/Agility thread here).

Reminds me of something I worked on once for an insurance company I won't name....

Integration Tests are a Scam - Quantified!

2010-03-23T19:46:00.000-07:00

JBrains is brilliant, no way 'round it. He wrote, directed and starred in this great indie, and if you want to see testing approaches dissected by the numbers, you'd better watch it.

Life/Death/Agility 5: Criticality and Changeable Requirements

2010-03-23T11:10:00.000-07:00

Milo Todorovich has pointed me to the granddaddy of FUD-on-Agile, Life/Death division: Balancing Agility and Discipline: A Guide for the Perplexed by Boehm and Turner. I'm trying to find a copy (IEEE wants $19 - hello, I'm broke!). I intend to post episode 4 of this series shortly, where I magically demonstrate the ultimate Agile approach to criticality, but meanwhile, here's a corrective to B&T - read the whole post:

The Boehm and Turner book is generally very informative and useful; I still turn to it as one of a handful of books that help me with Agile project management on a practical level. But here they seem to reflect the fear of adaptive development that was common in the early days. Organizations were reluctant to risk an important project on an approach that was, in their experience, unproven. Circa 2002 or 2003, the authors' guidelines made sense. In the 2006-2007 time frame, as the industry moves toward institutionalizing adaptive development at the enterprise level, we need decision criteria that are more in keeping with the realities of large organizations, and with the (now) proven ability of adaptive development to deliver high value and high quality results.

Criticality and Changeable Requirements

Because the criteria presented by Boehm and Turner are reiterated on Wikipedia , and will therefore be read by many, let's begin by re-examining them. First, I disagree with the authors' understanding of the criticality criterion. In our experience, it is precisely on projects of high criticality that adaptive methods have shone. Predictive methods tend to bog down in paperwork and procedure. By the time project teams have completed the heavyweight up-front analysis required with the predictive approach, the criticality of the proposed project may well have escalated to crisis proportions. An adaptive approach enables us to deliver results incrementally, and thus to begin to chip away at the critical problem almost immediately. Customers are usually more satisfied with that outcome than with the typical outcome of predictive projects. Increasing the chances of success is surely all the more important when the project is critical.

-- When to be Agile, by Dave Nicolette

More FUD on Agile

2010-03-11T09:44:00.000-08:00

The notion that the XP team leader tells the customer "Sorry, I can't commit to anything" is a bit of anti-XP propaganda that has been spread around. I can't imagine I'd get much business if I told people that.

-- Charlie Poole on Contractor Rate Based On Velocity - thread in Yahoo XP group

I haven't seen that, but it's not that different from the unbearable lightness of requirements.

Return of the Stakhanovites

2010-03-05T21:54:00.000-08:00

One of the realities of development these days is that our software teams have heterogeneous skill levels. Most teams contain a mix of veterans and rookies. There can be large differences in how seriously people take continuing education, too. My team, for example, has a member who just announced that he never learned any of the Java 5 features and so couldn’t work on areas that used generics. Considering that we’re now migrating from Java 6 toward Groovy, this team member is pretty far behind the curve.

-- Testing as Contract: A Modest Proposal

That would be a really serious problem for any team, but particularly one that hopes to be Agile.

I hope that - as the subtitle implies - the author is kidding when he suggests that the solution is for the "veterans" to write all the tests.

Life/Death/Agility 4: The Unbearable Lightness of Requirements

2010-03-04T10:19:00.000-08:00

Another in the ongoing miniseries, Life/Death/Agility.

I've seen this argument online and heard it in conversation: the reason you can't use Agile on serious projects like the Shuttle is that there are hard requirements that can't be changed.

Apparently it's a principle of Agile that the developers can change requirements.

I can just see a team explaining to a NASA manager, "We understand you thought you had this requirement, but you have to get over it and embrace change: because of the Agileological refactorization/cohesion principle, we're going to have to make the heat-resistant tiles out of Charmin."

Us:Language::Fish:Water

2010-03-03T19:36:00.000-08:00

Watching The Pluto Files, what struck me wasn't how upset non-scientists got about the "demotion" of Pluto, but the way astronomers, astrophysicists, etc. got all hot and bothered over it.

Whether Pluto is designated a "planet" is completely orthogonal to any practical/technical/scientific issues/research/hypotheses. Nobody's disagreeing about any non-linguistic facts about Pluto or the "legit" planets. This is exactly what people mean when they deride some controversy as "mere semantics" (and the opposite of what "semantics" is about).

We swim through this transparent, intangible medium all our lives, but every once in a while it hits us with a compression wave and we wonder - what the hell was that?

Career Pathing/ology - Getting at the obstacles

2010-03-02T09:51:00.000-08:00

Great analysis by Tim Ottinger: Career Pathing.

The biggest obstacles to Agile adoption are the built-in pathologies of industrial organization, the ways human motivations are manipulated. As Tim says,

In a typical career path, employees compete against each other for increasingly rare positions at higher levels. Esther Derby has written much about how this defeats morale and teamwork. In some situations one might find the "kiss up, kick down" strategy to be successful, where one panders to his bosses while sabotaging his peers and underlings. By political maneuvering, he is promoted (even "fast-tracked"), at the cost of productivity and harmony within the company. Distructive abuse of the career path discredits and defames the process.

The command-and-control model of organization (military-industrial complexity!) is echoed by waterfall methodology: an unchangeable truth is emitted from authority and governs all following behaviors (horizontal) and all subordinate personnel (vertical). This process gets its energy via the motivational patterns of the selfish gene/will-to-power/original sin.

More to come ...

Laziness in Moderation

2010-03-01T09:48:00.000-08:00

A good programmer has to be just the right amount of lazy. If he is too lazy, he will simply copy & paste his code and modify a few lines. In the end, his codebase will be unmaintainable, but he might be promoted due to the amazing lines-of-code (LOC) he has achieved. If he is not lazy enough, he will not shirk from doing something over and over again. Both character flaws (too lazy and too eager) result in code that is hard to maintain.
-- Heinz Kabutz, Generating Static Proxy Classes 2/2

Laziness is a subtle thing. I always figured "copy & paste ... and modify a few lines" was more work, and it strongly resembles "doing something over and over again".

Send in the enterprise clowns

2010-02-25T16:35:00.000-08:00

More on roles and titles: Architect or Circus Clown. Good points, except for the Eastern-Western angle. [I.e., it feels like Orientalism.]

British Airways - more life & death

2010-02-25T09:05:00.000-08:00

Both Croucher and emergn's consultants agreed there are areas of development where agile is not an appropriate choice or, in some cases, even an option. "Approximately 25% of our organization has already changed over to agile. And we are still working to grow that number; but in the process of growing that number we've identified areas where we are certain waterfall development is our only choice," Croucher said.

BA chooses to use the waterfall methodology in areas where there are a lot of requirements that can't be altered; where user communities are long established and the existing infrastructure is functioning at a level that can't be easily topped; and where major operational integration software has been built around an SOA foundation. "These are the 'big bang' areas where a tremendous amount of continuous integration is required and there is no room for discussion or testing of barriers," said Croucher.

-- from Where Agile development works and where it doesn't: A user story

Digesting this....

Requirements that can't be altered? How is that a barrier to Agile? Every project has constraints.

Ditto for SOA and, in general, legacy software that has to be accommodated.

"... the existing infrastructure is functioning at a level that can't be easily topped ...." This smacks of arm-waving. But if there really is a waterfall process that ain't broke, not only shouldn't you fix it, you should tell the world about it.

"... where user communities are long established ..." This is the one real issue: the difficulty of explaining the benefits of Agile to people who have major economic and emotional investments in the status quo.

This article is a specimen of a certain genre of "even-handed", "objective", "above-the-fray" punditry: sure, there's a time and a place for Agile, but NIMBY....

P.S. If you follow the link in that quote to "waterfall model", you'll see this:

Alternatives to the waterfall model include joint application development (JAD), rapid application development (RAD), synch and stabilize, build and fix, and the spiral model.

Something missing?

More life-and-death Agility

2010-02-23T20:48:00.000-08:00

The Rogers Commission found that NASA's organizational culture and decision-making processes had been a key contributing factor to the accident. NASA managers had known that contractor Morton Thiokol's design of the SRBs contained a potentially catastrophic flaw in the O-rings since 1977, but they failed to address it properly. They also disregarded warnings from engineers about the dangers of launching posed by the low temperatures of that morning and had failed to adequately report these technical concerns to their superiors. The Rogers Commission offered NASA nine recommendations that were to be implemented before shuttle flights resumed.

-- from Space Shuttle Challenger disaster - my emphasis.

Individuals and interactions over processes and tools - if Agile principles had been applied, the Challenger might be flying today. This makes the quote in my previous post even more hypocritical.

(Thanks to Sasha and Andy at Pathfinder for reminding me of this.)

Life-and-death Agility?

2010-02-19T11:58:00.000-08:00

-- picture from Space Shuttle Challenger Disaster

A software development life cycle model (SDLC) is a very important tool for successfully executing a software development project, and it’s important to choose the right approach for the right project. Some of the factors that would go into choosing the right SDLC model are:

Risk and Complexity of the Project – you would never choose a very agile approach for building the Space Shuttle software. The risk and complexity of the project is probably the biggest factor in choosing the most appropriate SDLC model for the project.
...

-- quote from Agile versus “Non-Agile” Development

When I read this, my first impulse was to say "just another old-line management type who doesn't get it". But now I have an uncomfortable feeling about it. Do I really believe that choosing waterfall development over Agile is a mistake - that the chances are good that it will come back to bite you? I do - I'm pretty sure I do.

Then do I believe it would be prudent or even possible to use an Agile approach to building shuttle software? Would I be a fanatic if I said yes? Would I be inconsistent if I said no? Don't I believe that Agile is the best way to deal with risk, uncertainty, unknowns? Can there be such a thing as "mission critical Agile", or are there some things that have to be done "the old-fashioned way"?

I certainly don't want to be a fanatic, but I think that if I don't believe that the best way to build the shuttle is with Agile, then there's something deeply wrong with what I do believe.

If I look at it from the standpoint of user acceptance testing, maybe the issue goes away. Just as skeptics say, "extraordinary claims call for extraordinary proof", we can say, "mission critical software calls for mission critical testing".

Suddenly this reminds me of how I overcame my fear of Ruby. I spent years fighting for type safety, and now you're telling me to start typing like a duck? I thought I could never trust a language like that. But then I realized that TDD is the answer - thank Dave for Rspec! - and I love Ruby to death.

If NASA doesn't have rock-solid acceptance testing, then it doesn't matter what approach you use - you can't trust the product. And if they do have rock-solid acceptance testing, you're free to pick the most efficient and effective approach, because your problem is now insulated by the solid walls of a black box.

And the most efficient and effective approach will be ... Agile.

This doesn't mean that things don't have to change: maybe we'd need a higher level of structuring of testing, above Fitnesse and XUnit and Cucumber and Rspec. Maybe we need automated requirements traceability, but with stories and tests and coverage testing, not BDUF and humongous documents.

But change is what we embrace, and risk is what we face - with

Courage!

[Bert Lahr pic from No Cowardly Lions Please!, which did not attribute source]

Because the world is round, you can't always say what you mean

2010-02-18T22:52:00.000-08:00

Mercator projection, from Map Projection. (Never knew they had such mathematical sophistication in the 16th century!)

Because the Earth is - sort of - a sphere, any attempt to project its surface onto a plane inevitably leads to distortion of some metric. Mercator projection is great for navigation because every straight line is an arc of a great circle, but it distorts the sizes of land masses as you move toward the poles. That's why there's not just one kind of projection - different maps for different apps.

One of the longstanding questions in linguistics is: why are there different grammars? For example, every possible ordering of subject, verb and object can be found in at least one language. Claims for universal grammar get pretty abstract.

Well, here's my two cents on it: because your mind is round, you can't always say what you mean.

Your neurons form a three dimensional network. Propositions/knowledge/ideas are tiny subnets of this network. Language is a great human invention, but with the significant exception of sign language, it involves a lossy projection of a three dimensional structure onto a one-dimensional medium.

And just as in the case of map projections, no matter how you do it, something is going to get distorted - some kind of information is going to be lost. This is why we have things like anaphora - for example, pronouns - that allow us to refer to the same "thing" in several different "places". In our minds, that thing is in one "place".

There's a dynamic tension in syntax between what's easy to generate and what's easy to understand. The syntax of any one language at any one time represents a dynamic equilibrium - tradeoffs in different dimensions. Syntax mutates and responds to a changing sociocultural environment because the relative perceived costs of different projective distortions change.

The same essential pattern applies to phonology - at any one time a language has a repertoire of phonemes that represent selection within two "competing" multidimensional spaces: the variety of sounds we can distinguish with our ears

(the domain of acoustic phonetics) and the variety of sounds we can make with our mouths

(the domain of articulatory phonetics). The tradeoffs are in terms of difficulty of distinguishing sounds vs. difficulty of generating them. And what determines those tradeoffs would have to be factors outside of language, as exemplified in various explanations of the the Great Vowel Shift.

When I first studied classical Greek, it amazed me that the people who spoke it apparently believed that it was so hard to pronounce a vowel followed by a 'b' sound that they had to put an 'm' between them, but the initial sequence 'pt' (as in "pterodactyl") was no trouble at all.

Anyway....

I'm sure this has all been said before, and I've oversimplified and overgeneralized up the wazoo, but it's a revelation to me, and I hope it will help me in my quest to ... uh ... do... whatever it is I'm trying to do....

[All illustrations are from Wikipedia]

More roles and titles

2010-02-05T11:49:00.000-08:00

I'm posting this posted reply to the post I posted about in my previous post because

Ron Jeffries is cool
Charlie Mingus is cool (is-not-was, even tho....)

Hello, Jerry. On Thursday, February 4, 2010, at 9:46:46 PM, you
wrote:

> To come up with what to code is the job of requirements analysts.
> So in most cases the wrong is not with coders but with
> requirements analysts. If requirements analysts were doing a good
> job, most, if not all, coders would be doing a good job.

Well, not according to how one does things in Extreme Programming,
in two regards.

First, the team includes what we call a Customer, who is ideally a
business person, not a "requirements analyst". The Customer, helped
by the Whole Team, expresses what is needed in three forms that I
call "Card, Conversation, Confirmation" , namely a small token to
remind us of what the "Story" is, and to use in tracking its status;
discussion of the need until the team and Customer understand the
problem; and tests, "owned" by the Customer but typically written by
the team, to determine that it works.

Second, your notion that coders would be doing a good job if
requirements were clear does not hold up in practice. In order to
write programs that actually work, we do need to know what to do. We
also need to know how to test that code, how to refactor it as our
understanding changes, and how to make sure that it keeps working.
It turns out that many would-be programmers do not have all the
skills necessary to turn "requirements" into viable "code", much
less the additional skills needed to be effective members of the
teams that create software.

Ron Jeffries
www.XProgramming. com
www.xprogramming. com/blog
Anyone can make the simple complicated.
Creativity is making the complicated simple. -- Charles Mingus

UPDATE - from another poster:

I would also like to suggest that the semantics in this case are important. Job titles like Architect, Requirements Analyst, etc. will pigeon hole people, and will cause folks to see boundaries that shouldn't exist.

Roles and Titles

2010-02-05T11:05:00.000-08:00

I just saw this post on the Yahoo extreme programming group (my emphases):

I came across a while ago sth like below:
"to code is easy; to come up with what to code is hard."
To come up with what to code is the job of requirements analysts. So in most cases the wrong is not with coders but with requirements analysts. If requirements analysts were doing a good job, most, if not all, coders would be doing a good job.

Would you think there's any chance the poster works in an Agile environment?

Earlier today I was phone-screened by a very friendly and smart guy from a consulting company who introduced himself as a technical architect. He said he does pure architecture sometimes and hands-on development other times. He later explained that Agile and Waterfall are per-client and per-project-lead options.

I've been a supervisor and a project lead. I was a manager once. I've been offered architect positions, but....

I'm a developer.

And a floorsweeper.

Tracking muck on the floor

2010-02-04T13:30:00.000-08:00

Kent Beck tweet: unit tests and source control are muck boots for the programmer's mind--you just don't have to worry where you walk

That's the thing of it! I'm screwing around with my Pragmatic module now - anything goes because I have enough Rspec coverage. I don't worry - I be happy.

Well, as much as I can under the circumstances.

Yet another frakkin' metaphor...

2010-01-28T17:28:00.000-08:00

Vintage Trek from 1984, courtesy of YakShaving

This is beautiful, brilliantly engineered by master craftsmen and insanely efficient, but it's not what I want to build. However, it's what I do for a living, and I have to exhibit enthusiasm...

Wright Flyer 1, courtesy of Wikipedia

This is ugly and somewhat makeshift, and is pretty lousy at its intended purpose. But....

Low-Hanging Fruit vs. High Risk

2010-01-27T10:39:00.000-08:00

Among my various misunderstandings of Agility is the algorithm for deciding what story/task/monster to attack first. It's either:

Go after the low-hanging fruit, get the process rolling with an easy win, warm up your tools and environment.

or

Jump into the biggest unknown first to reduce the uncertainty, start requesting external resources or services that you might end up waiting on, start researching something you may have a serious learning curve on.

With Sudoku, it's easy. I always go after the low hanging fruit - anything else would be absurd. Every cell filled in adds one more constraint to each of three cross-cutting sets - it lowers other fruit. There are no externals and no research.

With my linguistic ... uh ... hobby, it's radically different. I want to jump in and BDD me up some Rails, but I have so much linguistics to catch up on... The low hanging Rspec fruit is tempting, but biting into it won't bring me that instant knowledge of Good and Evil linguistic theories I need.

Who's More Agile - CSI-V or Leverage?

2010-01-15T22:52:00.000-08:00

The Leverage team is all specialists, no real diffusion of knowledge or skills. Of course, now they've got Seven of Nine, who's used to mind-melding....

The CSI-Vegas team all seem to know and do just about everything - including when to call themselves cops and when not to. (This struck me in the latest episode [two dead pseudo-hookers and a pseudo-Jekyll] when Catherine was running a major lab op to get fingerprints off a DB with neither of the med examiners in sight.) There have been learning experiences - when Greg (and later, Hodges) left the lab for field investigations.

Given that CSI-V has an aura of forensic science (in spite of - don't get me started - a major absurd fail at the MSI exhibit last year) and an imaginary lab to drool over, it does have some educational moments. Leverage has its post-hoc scam "reveals" - clever, but it's not in the same league.

Character personalities are front and center in Leverage and tend a bit more toward cartoons. CSI-V is darker (in all ways).

Toward the end of Dollhouse (just watched the penultimate episode), a team gels. The distortion here is that skills are "learned" instantly via technomagic.

Bones has the scientific aura of CSI-V and the rigid specialization of Leverage.

Okay - so I love these shows, but there's an issue here: pop culture inevitably provides role models. Are they good models for real work teams?

P.S. What is it about the techies' names on these shows? Bones has Hodgins (although Angela's the software geek), CSI-V has Hodges (who seems to have taken over Archie's responsibilities), and Hardison on Leverage is played by an actor named Hodge.

P.P.S. CSI-NY is a yawner and CSI-Miami is an obscenity - I won't even link to them.

...

Where have I been lately? Looking for an "engagement" and Railing on my linguistics hobby.

Rails and (unrelated) Rants

2009-12-20T12:11:00.000-08:00

Rails TDD Boot Camp last Monday thru Thursday at Obtiva was great. Good small group (Alan, Ashutosh and me - should have got their email addresses!), good instructor (Andy), good coverage. I'm ready to create websites to do just about anything.

(Unrelated rant:) This is so true: Aliens, Elves, and the Politics of Utopia.

That's all. Have a nice holiday break, and don't wear a red shirt....

[Oops, posted this on the wrong blog at first, but it's okay, nobody reads that one....]

Thanks to Andy...

2009-12-06T21:37:00.000-08:00

... I'm running on Rails at last.

I had problems getting Rails talking to a database - either Sqlite or MySql. I figured since I'll be at Ruby on Rails TDD Boot Camp next week, and Andy Maleh's teaching it, I could ask him for help. And help he did: we spent over two hours at the Red Eye Cafe (4164 N. Lincoln - free wifi, free parking!! and best of all, AC outlets by our table!!!) working through the combinations of versions of MySql, the mysql gem and 32/64 option in Snow Leopard. Without going into the boring details, the combo that works is MySql 5.1.41-osx10.5-x86_64 and mysql gem v. 2.7 installed with x86-64 support. The kernel default in Snow Leopard is set to 32 bits because I haven't upgraded VMWare Fusion to support 64 (they want $40 - I'll try VirtualBox before I spend that!). Other combinations cause various errors in either rake db:create or rake db:migrate.

Thanks again, Andy!

Baby in the Bathwater

2009-12-06T13:36:00.000-08:00

Yesterday's post started on a relatively optimistic note and ended less positively. That was an overreaction, and it reflects a pattern that appeared earlier in the spike. I test-drove some classes, decided they were not going to solve my problems, and abandoned them to go in a different direction. But later I discovered that there was a good use for them within the new structure.

In the first case, I had been working through the char-by-char element extractor class. I eventually decided that the complexity issues I was having with it meant that it was a cul de sac of premature generalization, premature concern with implementation details and/or overengineering, and decided to drop it in favor of the W1913 element extractor aimed at a very specific story. In the course of test-driving the latter, I found that it could use the element extractor exactly as it was.

Now it seems to me that the pattern repeated. When Dave paired with me Friday, we at first tried to deal with some of the complexity by refactoring. But then he suggested a much simpler approach to extracting entire dictionary entries by scanning an entire file with regexps. The result was beautifully simple and fast, and so at the end of that day's post, I declared the more complex code I had developed to be floorsweepings.

But now I think that was premature - there's a baby in that bathwater (so much for metaphoric consistency with floorsweepings!). (I don't mean to imply that Dave was claiming his solution would work for my other stories - in fact he said it wouldn't.)

I'll approach this from another direction now - the distinction between semantics and pragmatics.

Something that annoyed me when I first saw the W1913 was the mix of visual and semantic markup. Headword, PoS, sense and definition tags were intermixed with page break tags, , , etc. - purely visual markup. (These days tags like are deprecated in HTML in favor of things like <emph>. The markup in W1913 was inserted in 1999.) The bottom line for W1913 is that it supports two radically different purposes: the original intention was to be able to reproduce a book, but my interest is in extracting dictionary entries for other kinds of processing. This is a pragmatic distinction based on differing purposes - closely related to the distinction we make in application development between models and the processes that use them (services, views, controllers, etc).

Semantics is simpler than pragmatics because it's context-free. Using regexps to extract W1913 entries makes perfect sense. However, the structure inside an entry is more complex for several reasons. One is that page breaks and visual tags are noise from the current perspective. But more important - there's context. Here's an example from the A-B file:

<hw>A*base"</hw> (&adot;*b&amacr;s"), <pos>v. t.</pos>[<pos>imp. & p. p.</pos> Abased (&adot;*b&amacr;st");<pos>p. pr. & vb. n.</pos> Abasing.] [F. abaisser, LL.abassare, abbassare ; ad + bassare, fr.bassus low. See Base, <pos>a.</pos>]<sn>1.</sn> <def>To lower or depress; to throw or cast down; as, to abase the eye.</def> [Archaic] Bacon.

There are four occurrences of the <pos> tag here. The first one follows the headword - it says "abase" is a transitive verb. The next two refer to inflected forms: "abased" is both an imperfect and past participle, while "abasing" is a progressive and a verbal nominalization. (It's not clear why these should be present in the entry at all, seeing as they are perfectly regular inflections.) The fourth occurrence is in an etymological section: the word "base" in the sense of "crude" or "vile" is an adjective.

What's relevant about this example from the perspective of extracting PoS information is that in the first and fourth occurrences, the form to which the PoS element applies precedes that element, whereas in the second and third case, the form follows the PoS. The markers that would help us distinguish these cases are not tags, but square brackets and possibly the word "See", or the language markers ("F." and "LL."). The simple line separating data and metadata (text and tags) is blurred. (To add to the complexity, there are many entries where the headword has no PoS tag at all.

Semantics is about independence of context and history - that's why it's simpler than pragmatics. Since pragmatic information is progressively added to and removed from context (scoped), I suspect it's easier ultimately to use a state machine to parse it. My first crude attempt at this needs a lot of rework, and certainly applying the regexp scan makes the problem a lot simpler - it removes one layer of structure, so I can concentrate on intra-entry parsing.

Spike Day 5 - Last Day

2009-12-06T08:09:00.000-08:00

(See New Broom for an explanation....)

12/4/09

Priorities

This is the last day of the spike. If I’m going to proceed as if this were a normal project, there are several options: create the Entry Builder and the Orchestrator, and finish the Entry Extractor.

The Entry Builder has to extract PoS and sense info from the entry string and construct a new object, a lexicon entry optimized for the yet-to-be-specified parser.

Actually, there’s more to it than just distinguishing senses. There’s a morphological issue: the parser will be seeing inflected versions of words. Some of them, like “walks”, “walking”, “walked” and “walker” can be mapped by general rules to their root words (“walk”) – these are the “regular” nouns and verbs, and typically the rules for encoding and decoding them would be in a separate component. Others, like “children”, “knives” and “fish” (plural) are “irregular”, and these forms are always in the lexicon.

The Orchestrator is essentially for setting up the pipeline: wiring up the queues and transformations/filters/processors. (Think Mario Brothers – or am I still missing Spring too much?).

The Entry Extractor needs a few more rspec cases.

Okay – given that there are outstanding domain issues on the Entry Builder, the Entry Extractor work is pretty straightforward, and this spike is more about learning how to use Ruby constructs, I’ll proceed with the Orchestrator and wire up the pieces I’ve already got.

First, some refactoring.

The EOS (end of source) string I started out using – “***EOS” - is problematic because the asterisks need to be escaped for it to be used as a regexp pattern. There’s no need to have more than one of these things, so it can be a constant and not have to be passed into to each processor class. I’ll set up a Pipeline module for all the classes to include.

The pipeline subfolder I created was primarily for seeing how require paths work – and it still confuses me a bit.

Project Structure and Require

Now I want to move some classes around and see how Git and TextMate interact when I do that.

Created a processors subdir and moved all the processor classes into it, along with their rspecs. (Should there be a separate parallel test hierarchy ala Maven?)

Jay Fields’ Thoughts: Ruby Project Tree does advocate that separation, so I’ll follow his lead. I won’t take his suggestion re what Micah’s guidelines on Ruby require calls “require farms”.

I’d like some Montrachet with this Pinot Noir: there is such a thing as “require hell”! I was sizzling in it – until Joe Banks (and Dave Chelimsky’s pre-release Rspec book PDF) bailed me out. It took quite a while to clean it all up. In the process I realized that Rake is something I still need to understand.

Spike End: Lessons Learned

What I learned is what I still need to learn.

Interestingly ambiguous sentence – there’s a contradictory interpretation: “there’s a set of things that I still need to learn and I actually learned them (so I don’t still need to learn them, but then ....)”. What I intended was: “there’s a set of things I still need to learn and I learned what that set is”.

The biggest thing I need to learn is to unlearn what my years in the waterfall business made second nature: implicit BDUF. What’s wrong with BDUF is that it’s almost always the wrong design. I had a graphic demonstration of this when Dave H. went over my project and found a simpler way to do just about everything with regexps and hpricot. The faulty assumption – that BDUF habit - was that I needed to work with character streams.

(Actually, this character-by-character parsing happened in a class from the earlier phase – the Tag Extractor – which I essentially abandoned for the right reason when I started on the W1913-specific classes.)

If I can bring a whole dictionary file into memory, Dave’s solution will do the whole trick. None of them is bigger than 10 megs, so it could work.

Tried it: sure it works! – and it handled the “chunking” of entries in the A-B file (~5 megs) in less than a second.

There were two occasionally conflicting goals for this experiment: learn Ruby and craft a project. I learned a ton of Ruby, and a lot about what I still need to learn about the craft. The project code itself is not particularly usable for practical purposes –

Floorsweepings.

Spike Day 4

2009-12-03T20:54:00.000-08:00

(See New Broom for an explanation....)

12/3/09

Priorities

Feels like I’ve been yakking too much here and not coding enough, so just coding is a priority for today.

Let’s see how fast I can get a pipeline going without agonizing over the likelihood that there are better Ruby ways to do this.

I’d like to set up a git repo to keep a baseline of the extractor project, so I’ll timebox that. One hour -too much? I really want to use git, so one hour it is.

Maybe later I’ll talk about yesterday’s two interview questions (pairing, evaluating tech) and last night’s dream (grammar school).

Git ‘er did

Setting up the Git repo took 25 minutes from scratch. Admittedly I cheated, having experimented with Git before. But today I started by googling and used this page: Git - SVN Crash Course, because I’m a longtime Subversion user. Everything I needed to get started was there. The only thing it didn’t cover was .gitignore – and that was what I had had trouble with before. I wanted to ignore the logs folder. So I googled that and gitignore(5) did the trick.

The moral of this story is: to learn something new, see if you can start from where you are (pragmatic) rather than from some ideal image of where you want to be (semantic).

So I’ve tagged the baseline, and now I’ll start pipelining. But I want to remember to look at the code Dave recommended as an example of good Ruby style. It’s on github, so I’ll pull it when I have time. I’ll put that on a card (thanks again, Milo).

Pipeline

Start with a story:

As a ... what? ... parser developer? ... I want to extract entries including PoS-by-sense from the public domain Merriam Webster 1913 dictionary, available online in a set of semi-marked-up text files for eventual feed into a lexicon builder.

Decompose story into a set of tasks as follows:

Read files.

~~Split into buffers.~~

~~Queue buffers in sequence.~~

Extract entries as strings.

~~Queue entry strings in sequence.~~

Build entry objects with PoS, headword and sense exposed.

~~Queue entries.~~

The struck-thru buffer and queue tasks were premature. They should emerge from a process of organizing the primary tasks into objects.

Objects:

FileReader

EntryStringExtractor

EntryBuilder

Orchestrator

This isn’t completely satisfactory but it’s a start.

I should have done this the first day!

I could create CRC cards (does anybody still use them?). I haven’t done this in years but it’s worth trying. I’ll write them here and then on real cards:

FileReader

Responsibilities

Accepts a sequential list of files, a queue, an EOS marker string and a buffer size.

Opens files in sequence.

Reads files into buffers in sequence.

Puts buffers on queue.

Puts EOS marker on queue after last file is processed.

Collaborations

File

Queue

EntryStringExtractor

Responsibilities

Accepts an input queue, an output queue, entry start and end patterns, and an EOS marker string.

Extracts entry strings from input buffers.

Puts entry strings on output queue.

Passes EOS marker to output queue.

Collaborations

Queue

ConstituentExtractor

EntryBuilder

Responsibilities

Accepts an input queue, an array of patterns for element extraction, an EOS marker and an output queue.

...

Wait a minute

This is starting to smell like BDUF. I don’t know where to draw the line, but the EntryBuilder definitely needs more thought – extracting multiple PoS-sense objects from one entry is not so simple. I’ll probably need to inject one or more extractors into the builder, and they’ll be more complex than the tag extractor, although they can use it.

This won’t be finished during the spike.

I’m realizing that it was unrealistic to expect to be able to come up with something anywhere near as elegant as the Bowling Game Kata – and that’s been in the back of my mind from the get-go. What I’m trying to do here is nowhere near as well-defined as a bowling game.

There’s also the painful tension between working with a very specific source like W1913 and wanting to end up with something reusable. I think the best move is to bite the bullet, go with the specificity (pragmatic!) and have faith in the process: when there’s another source to be plumbed, it should be possible to externalize the specificities thru refactoring.

BDD : pragmatics :: BDUF : semantics.

So I’m definitely going to stop now, go with the FileReader and the EntryStringExtractor (which I hereby dub the W1913FileReader and W1913EntryStringExtractor) and see what things look like when I get to the point of needing an entry builder.

Dependency Dejection

Another learning bump – after months of intensive Springing, I have to get used to the absence/irrelevance of dependency injection in Ruby. I wanted to test the file reader by injecting a mock file object. Thanks to a chat with Craig D., I’m now using StringIO for that (also covered in TWGR).

Githubbed?

Created a repo on Github:

http://github.com/GHogChi/FloorSweepingsCraftsmanSpike

Now I have to push to it. L8r.

Progress

Got the file reader working to match the story.

Got the entry extractor to the point of a non-trivial failing test (git rev 1ab471ac08be62b076d29cc5a7664f370f05b83b). Now I have to plug in the element extractor I created previously. But time is short – should I try to push the repo or finish the story?

Pushing the repo is for the blog readers – so far they are few and local, so I can show them the code locally. The push can wait.

Got deadlocks, have to continue tomorrow.

Git

I think the push worked. Here’s the public clone URL:

git://github.com/GHogChi/FloorSweepingsCraftsmanSpike.git

Oh yeah..

About the two interview questions and the dream:

Is pairing productive?

I hope I was clear about it. What I should have said is that notwithstanding the studies that show it ultimately saves time by catching problems early, it establishes a pattern of open information flow within the team.

How do you evaluate technologies?

What I hope I/should have said: after all the research/googling/book reading, it's good to do a test drive and give yourself a chance to get enthusiastic about it: as a developer, you'll have to live with it.

Dream

I dreamt that as an adult, I had somehow applied to an elementary school and been accepted. I wanted to get back to basics. I ended up having a conversation with some teachers (in the teachers' lounge?) about our graduate degrees.

Okay, it was weird - it was a dream, after all. But I think part of it is a reflection on the back-to-basics nature of this Craftsman Spike.

Spike Day 3

2009-12-02T15:52:00.000-08:00

(See New Broom for an explanation....)

12/2/09

Priorities

Finish the extractor BDDing or start on pipelining?

Dave’s keeping the definition of the “Craftsman Spike” open is a blessing and a curse. It’s forcing me to prioritize across “meta” boundaries and confront the local existential question: why am I here?

Prioritization is part of the craft, so in wrestling with these issues, I am learning something important.

The story-level cycle (think about the story, run RGR cycles, repeat) requires a way to guarantee that the “thinking algorithm” will halt at a reasonable point, avoiding “analysis paralysis” and over-engineering. (In fact, the same issue comes up within the RGR cycle itself – cf. Red-Green-Refactor.)

Rereading that post by James Shore reminds me that I still need to work seriously on keeping the coding short in the RGR cycle. That first fiasco with the TagExtractor is a case in point. In fact, this is another example of the halting problem. Given that emitting only short bursts of code – baby steps - is a big priority, it points toward finishing the extractor story.

But another priority here is learning Ruby, which I’m interpreting as “getting some experience with all the constructs that are new to me as a Java/C# developer”. From that perspective, these extractor stories are like micro-spikes that don’t have to lead to production code, just proof of concept.

It occurs to me that I can use this log/blog/diary as part of the cycle in a more granular way as a check on concision.

Interrupt

In the spirit of making this more real-time (and probably more boring to folks reading it – hmm, should I be tweeting?), I'll mention that I just got an email from Andy M. about my Rails getting-started DB problems. He’s asking for my database.yml and Rails logs. (Embarrassing that I never looked at them....) So hold on while I go look for those.

Interrupting the Interrupt

Todd just cranked up Pandora – his blues station – and I didn’t recognize Howlin’ Wolf. That’s gigantically embarrassing, but I can’t let it stop my march toward craftsmanship.

Back to Priorities

If I want to finish the constituent extractor (CE), there are two stories I haven’t dealt with. The easy one is – extract multiple constituents from one string. The hard one is figuring out how to deal with EOS, because in the current rspec context, the end_matcher is the same as the start_matcher – i.e., the beginning of a W1913 entry is a headword element, and the only way to find the end of it is to find the next headword.

So I’ll start with the easy one and concentrate on concision.

It occurs to me that I could be providing zips of the current state of the code for anybody crazy enough to want to follow this stuff in detail.

Don’t see how to do it with Blog*Spot free hosting, although they will host images.

Stupidity Interrupt

How the hell did I ever decide that the initials for Red-Green-Refactor were “RGF”? Sheesh! It’s RGR from now on. But I won’t go back and correct it. (Stop laughing, Milo!)

[UPDATE: I lied - I went back and fixed it. No point in confusing the reader.]

Back to CE

Going with the easy one first. Shore wants me to restrict myself to five lines of code per cycle. Let’s see if I can do that. Should I prefactor? (Hmm, that word seems to have turned into something else since I heard Paul P. and Micah M. using it.) I think it will be harmless to prevent duplication by making my headword matcher an instance variable and moving the definition into a before(:each), although Craig D. instilled the maxim “see the duplication before you remove it” in me.

(Okay, I’ll admit it – I really miss Eclipse’s (and Visual Studio/ReSharper’s) instant highlighting of errors.)

Another prefactoring – extract a validation method for the results.

Leaning over backward to keep it under five lines: wrote this very long one:

results = ConstituentExtractor.new(@headword_matcher, @headword_matcher).extract('AxByC')

(If it works, I’ll go back and verify that Ruby won’t let me break it before or after the ‘.’.)

Oops – I needed the offset argument in the extract call. Now it’s the “shade of red” I wanted, and the test is only four lines long. (Just checking the size of the results array for now – expected 2, got 1.)

Extracted the extraction code to a private method. The extract method calls it in a loop and increments the offset as long as there’s a match.

Ran the test – good news, bad news: the single extraction test is green, but the double extraction is still red. I’m inferring that the problem is possibly a one-off in the offset handling in the loop.

Interrupt

Milo T. IMs me. Interesting convo. Some of it will show up here. He’s reading this blog. Suggesting I use cards to put some of the technical overthinking on the stack to keep it from slowing me down – a good idea for production, but this Craftsman Spike is rehearsal – speed is not the issue here, and overthinking is part of the process.

But Milo’s right, kids – kards are kool!

Back to CE

Wait – there is no loop yet! No wonder it only found one constituent.

Set up a begin .. end until loop. Oops – infinite - had to force-quit TextMate!

Meta: Agile Athletics

After the IM with Milo about metrics and micrometrics, I realize I’ve thought about Agile development as an athletic activity for a long time – it goes back to Kent Beck and the name XP. So the Craftsman Spike is like training camp – not necessarily just boot camp, but a refresher.

Back to CE

Corrected the offset calc, and all is green!

Refactor: deleted the debug console dumps. Could probably compress the extraction into fewer lines, but I want to move on.

Now the hard one – dealing with the end of the source text (EOS).

Realized I could put the project into git – that would be the simplest way to track it through time. Still a little learning curve there – last time I tried git I had problems getting it to ignore some things.

Okay – trying EOS. I hope it’s as simple as an alternation (‘|’) in the end matcher. Worked in rubular anyway.

Got to green on the EOS test, but boy does this need refactoring! Passing in a couple of procs helped me learn about procs (certainly a spike goal), but the justification for it (duck typing: you could pass in anything as a matcher as long as it returns something that quacks a bit like MatchData, in case ordinary regexps aren’t enough) isn’t cutting it at the moment, mainly because so much of MatchData is being used now that EOS is supported. What’s worse: the extractor is making assumptions about what’s captured in the regexps.

So I’m going to pass in three regexp patterns: constituent start, constituent end, and source end. It was another case of premature generalization – regexps are fine for my current story. I have to have faith in the process: keeping the code DRY and OO-clean will make it easy to generalize in future if necessary.

Let’s see if I can do this refactoring in baby steps without breaking tests.... With Java constructor overloading it might be easier, but too bad.

Okay, after a couple of false steps, it’s done. No more procs – but at least now I know how to use them.

Noticed something nice (it may happen with Java too, but I don’t remember seeing it, maybe because Java code is more verbose): every time I refactor, the code gets more compact. Certainly true of the rspec class. In the extractor class, the private extractor method is 18 lines long, which seems like a lot, but the algorithm is kinda complex and I used intermediate variables to keep it a little clearer as to what’s happening. There’s one comment, and it regrets itself:

# (smelly that this needs a comment) following line handles the normal and EOS cases:

end_match_index = end_match.captures[0] ? end_match.begin(1) : end_match.begin(2)

Anyway, the exposed surface of MatchData is not all that transparent.

So – pipelining?

Pipelining again

I’ll start from the outside and maybe get into some Ruby file IO.

...

Just got sidetracked into looking at issues with IO and String classes and encoding. Can’t expect to come up with the ultimate efficient solution now – just want to concentrate on reading in files, pipelining, extracting, etc.

Will get to it tomorrow.

Spike Day 2

2009-12-02T07:19:00.000-08:00

(See New Broom for an explanation....)

12/1/09

Procs and Duck Typing

Picking up from yesterday: got the first rspec for the CE passing – no match returns empty array. That establishes the infrastructure of the class but with a minimal commitment to anything else. Now I have to decide exactly what should come back as the result of extraction.

How about: start and end offsets and the match for the tipoff. In the case of W1913, that would be the headword element: someword. In the case of the PPP (Post-Platonic Parser – TBE [To Be Explicated]), that would be a matter of matching a word like ‘the’ (the tipoff) and the noun phrase it introduces (the rest of the constituent).

Or should it include the entire substring that represents the constituent? To say no at this point would smack of premature optimization. So I’ll go for it.

Should it be an object, a hash or an array? I’m leaning toward an object. I’m not sure that’s the simplest thing that can possibly work, but it seems cleaner. I’d rather create an object I don’t need and eliminate it later than have to find all the places I’d have to change a hash or an array to an object. (I’m afraid I’m still thinking too much in terms of static typing rather than duck typing, and maybe I’m overly concerned about the lack of refactoring tools.)

Smelly

Got to green on it “should return the proper constituent on a match”, but it took too long and involved too much code. I learned a good chunk about procs and MatchData, but I’m not happy with my BDD process or the resulting code.

Part of the problem seems to be premature generalization – trying to implement two stories at once. I should have stuck with the W1913 entry extraction. Especially when I’m working with code constructs I’m not familiar with.

And there’s something smelly about the Constituent class I created, beyond its being just a transfer object. It exposes the tipoff string and the full constituent string, which is okay, but it also exposes the offset of the constituent in the string passed in. That datum is irrelevant to the actual use of the object, and it ended up there for two bad reasons. The first is that I was thinking ahead to the problem of buffering the multiple files that make up W1913. The second is some interference from thinking about citations (see yesterday): I want a citation object to track the location of its target string within a source object. (I want to track offsets in the PPP as well.)

Does the fact that I’m wearing two hats at once (customer and developer) have something to do with it? Having a real dialog would help to clarify the issues (minimalist version of crowdsourcing – N heads better than one). Similarly – I’m not pairing.

Wait – I’m beating myself up unnecessarily here (stop grinning, Milo!). The offset is not completely illegitimate – it really is part of the story - maybe it just needs to be packaged better. There are local/contingent offsets within the strings (buffers) that are passed into the extractor, and there are persistent and meaningful offsets in the sources (files).

And let’s face the generalization issue head-on. There are lots of situations where we do this kind of thing: screen-scraping, data-mining, etc. No way this puppy can handle all of them. Generalization is way premature. If I end up with multiple classes that do very similar things, and they’re all good OO citizens, generalizing should be a clean refactoring. So – back to the W1913 story.

Pipeline?

A new thought: what if all these extractors could be pipelined? E.g.:

W1913 files | entry extractor | PoS extractor | lexipedia updater [TBE]

(I would have used “=>” there, but it would collide with Ruby notation, so I went for the *nix operator instead).

Threads communicating via queues - not only could it work, and fit into the bigger picture, it gives me a chance to work with threading and file IO in Ruby.

But first...

... a bit more work on the constituent extractor.

A thought about offsets: if I take the citation/source concept seriously, the current constituent extractor is working on essentially transient strings. So the source object would have to reflect that, and the citation object would just be an offset and a reference to the source. The constituent extractor could easily inject citations into constituent constructors. For pipelining, the thread that feeds the extractor could use the returned offsets in managing its buffers.

Spike Day 1

2009-12-01T14:52:00.000-08:00

(See New Broom for an explanation....)

11/30/09

Code

Went back to the POS extractor project. Moved Element from element_extractor.rb to its own file. Tried to run the rspec from TextMate. Got exception:

/Users/Tom/.gem/ruby/1.8/gems/rspec-1.2.9/lib/spec/runner/options.rb:282:in `files_to_load': File or directory not found: nt_extractor_spec.rb (RuntimeError)

Somehow the rspec file name is truncated.

Tried running a different rspec file. Name truncated – in both cases it’s the first five characters.

Tried running the element extractor test from the command line:

./element_extractor.rb:2:in `require': no such file to load -- log4r (LoadError)

Have to decide which issue to pursue. I want to be able to use TextMate, so ultimately that one needs to be resolved. I also want to understand how to specify the load path for Ruby.

Googling the TextMate problem didn’t lead to a solution (nobody else seems to have this problem)....

Meta: Baseline Meeting with Dave H.

... but talking to Dave did – rubber-duckily! I had opened the project in TextMate at the wrong level – once I opened it at the folder level, everything worked fine.

Also, he explained the command line problem: I was trying to run the rspec tests with ruby rather than spec.

Dave saw that I was not using Rspec mocks correctly (could have just used an Object.new) in creating my “character stream”, and the need for a ‘fake’ class emerged from the convo. I suspect that I would eventually have been driven to this conclusion by duplication across tests, but part of the goal of this spike for me is getting to a “Ruby state of mind” where these things occur to me sooner. As a beginner, I’m bombarded with lots of different issues while trying to write code, and only when some operations become “second nature” will I have enough mental cycles available to see things like this early.

Efficient use of resources is a major goal, one I suspect should be part of the apprenticeship process. The learning process is like meta-RGR (red-green-refactor): I get something accomplished, but then I need to review how I got there and see how my own thought processes can be refactored, made DRYer and simpler.

The issue of whether to get into Rails or continue with Ruby seems to be at least temporarily resolved in favor of Ruby after the convo with Dave, both for technical reasons (no immediate solution for the Rails problems) and for reasons of interest (linguistic) which seem to trump the career-oriented argument for Rails. I’m a starving artist!

Dave also turned me on to WordNet (http://wordnet.princeton.edu/).

Meta: Word for Mac destroys Universe

I was just about to enter something here (following section) [I started writing this blog in Word for Mac] when I made the mistake of using Cmd-W to close a Find dialog. My MBP rebooted!

Lesson: if it hurts, don’t do it.

Fallout: Firefox crashed, leaving its profile locked. Have to clean that up to get back to my bookmarked pages.

How to deal with distractions/tangents like this without losing momentum? One answer would seem to be: find the simplest fix that can possibly work, but postpone the refactoring (i.e., preventing the problem). Put it on a card!

Finally got Firefox back after having to delete both .parentlock and places.sqlite from the profile folder. No simpler fix – I needed all the bookmarks! (I had to use Safari to find this info – glad it’s there!)

Changing stories

BDD’ing the Pos Extractor has led me to some new stories.

I need to identify the beginning and end of an entry in W1913. This goes beyond element extraction with tags. Entries begin with the (headword) tag, but the headword element is just the word name. The only way to identify an entry is to find the beginning of the next one.

Now I’m thinking that’s just what I need for analyzing constituents in text in general: a constituent extractor. It would have to operate on strings, not character streams, because it potentially needs to back up more than one character if it’s a matter of some kind of pattern match for the beginning of some other constituent. In the case of a W1913 entry extractor, that would mean finding another headword start tag or EOF.

Can I use Ruby regexps for this? They would have to return the index of the beginning of the match. Or ... maybe it could be a Lispish thing of peeling off the head and leaving the tail.... The index isn’t the essence of the story: moving through the string is the key. As long as the regexp can return two matches, the constituent and the tipoff (e.g., start tag) of the next constituent, no index is needed.

Never mind! This is exactly what I need:

“Where match and =~ differ from each other chiefly is in what they return when there is a match: =~ returns the numerical index of the character in the string where the match started, whereas match returns an instance of the class MatchData....“

-- The Well-Grounded Rubyist by David Black – Section 11.2.2, p.322

(This is a great book - the best programming language intro I've ever read!)

Implish thought:

... as in “implementation” and “impish”.

Could operate on a char stream by extending buffers until a match or EOF is found.

Seems best to proceed by ignoring this, working with strings, and having faith in the process – i.e., that using hygienic practices will allow for easy incorporation of streams later.

Time out to read TWGR on regexps and procs.

The goal here is for the PoS extractor to get a series of complete entries from its input, then extract PoS info from each entry. A constituent extractor would be used to get the entry.

This gets into the other objects I want: citation and source identifier.

Citation is an object that specifies a string within some persistent source entity: a file, a blob in a DB, etc. , but does not actually contain the string. It should be able to produce the string on demand.

A citation would hold a reference to a source identifier, an object that offers one or more ways to access the source entity: URLs, filepaths, DB queries. I’m not even going to try to imagine what “access” will turn out to mean in implementation terms.

Is this BDUF again? Not if I BDD the constituent extractor without prejudice. If it doesn’t lead inevitably to a citation, so be it.

The reservoir

The above should have been a matter of thinking about possible designs without committing to anything. Here I’m following the concepts of one of my mentors, the actor, director and teacher Steven Ivcich.

He has a very nonstandard approach to preparing for performance. It involves a lot of near-random improv and playing with the script. His term for this is "filling up your reservoir". The idea behind "classical" preparation and rehearsal for an actor is that you are locking down the character and all your moves before you ever perform. Steven's approach is much more agile: every performance involves new information - new audiences, new energies, new twists on the interaction with other actors. You develop the reservoir as a way of preparing to respond to unexpected developments. In other words, no Big Design Up Front.

To bind the analogy: the RGR cycle is the performance, and thinking about the domain and possible technologies before going into the cycle is filling the reservoir.

BDDing the Constituent Extractor

I’m thinking that the CE is initialized with two procs for matching the beginning and end of a constituent, each returning an offset (maybe they just run a regexp, and maybe not), and it replies to messages containing a string and an offset with an array of strings – no, constituent objects, each of which has the offset of the beginning of the constituent and a length. If the CE finds the beginning of a constituent but not an end, the length will be nil or infinity.

This is bothering me. Already too implish? Pushing toward the constituent object being a citation?

More Run-up to the Spike

2009-12-01T14:39:00.000-08:00

(See New Broom for an explanation....)

11/29/09

Got the EE into reasonable shape. Instead of proceeding on the PE, I’m going thru the Rails startup in Agile Web Development with Rails.

Set up the demo project. Need to make MySQL the DB.

Problems with database.yml: can’t use mysql – rake db:create throws a not-very-informative exception:

Couldn't create database for {"timeout"=>5000, "username"=>"root", "adapter"=>"mysql", "database"=>"railsdb.myapp", "pool"=>5, "host"=>"127.0.0.1", "password"=>nil, "socket"=>"/tmp/mysql.sock"}, charset: utf8, collation: utf8_unicode_ci (if you set the charset manually, make sure you have a matching collation)

Googled it. None of the proposed fixes work.

Tried creating the database manually. Now rake db:migrate throws an exception:

uninitialized constant MysqlCompat::MysqlRes

Googled it. Again, none of the proposed fixes work.

Followed advice from my Ruby help-line (Craig D.): use the Rails default DB sqlite3.

Now rake db:migrate succeeds, claiming it created the “users” table. After generating the scaffold, I try running the server from TextMate. It complains:

RuntimeError: Please install the db/sqlite3 adapter: `gem install activerecord-db/sqlite3-adapter` (no such file to load — active_record/connection_adapters/db/sqlite3_adapter)

So I try sudo gem install activerecord-db/sqlite3-adapter which results in:

ERROR: could not find gem activerecord-db/sqlite3-adapter locally or in a repository

I give up for the evening, but I’m thinking: if you’re going to throw a runtime exception that won’t be handled by code, you should expect that it will end up in a logfile and some human will have to try to figure out what happened. So why not have a mechanism that dumps not only a stacktrace but the context – like all variables in scope (locals, method arguments, instance variables, etc.) at each stack level? Better to have way too much than way too little.

Run-up to the Spike

2009-12-01T08:27:00.000-08:00

(See New Broom for an explanation....)

11/28/09

I’ve been writing Ruby using TextMate and Rspec for a couple of days now.

Yesterday I TDD’d a TagExtractor specifically for extracting PoS (Part of Speech) info from W1913 (the Project Gutenberg semi-marked-up Merriam-Webster dictionary from 1913). Strayed from the path – got into a very complicated parse method with multiple flags.

I decided to step back and approach the problem from the direction of “pure story” – i.e., let the story drive the test and let the test drive the implementation decisions. This resulted in BDDing of the
PosExtractor, which will take a character stream and return an array of triples: {:word, :sense, :PoS}.

Now there’s a dialectic between the PE (PosExtractor) and the TE (TagExtractor) which is starting to drive the TE toward a simpler and hopefully more Ruby-like implementation, using Element objects that consume from a character stream.

Starting with the TE was the first mistake: a bottom-up approach locking into an implementation detail that should emerge from BDD. BDD done right prevents BDUF; TDD can encourage it. Starting from the nouns and verbs of the story makes possible a progressive decomposition of the
story into objects and functions.

Required log4r to debug TE.

Created new class ElementExtractor (EE) from TE to eliminate the find_all_tags noise – BDD’d it.
Now the Element class, which was just a transfer object that accumulated content, has evolved under BDD pressure to detect its end tag and mark itself complete, which simplifies EE.

New Broom

2009-12-01T08:20:00.000-08:00

I'm starting this blog to track my software apprenticeship - for an explanation of that term, see Apprenticeship Patterns: Guidance for the Aspiring Software Craftsman by Dave Hoover and Adewale Oshineye. In fact, buy the book, so you can understand why somebody who's been in the software development biz since computers ran on kerosene would want to be an apprentice, and whence the title of this blog.

I'm going to start off with some postings from my Ruby diary. I'm at Obtiva this week, participating in a "Craftsman Spike". Instead of trying to explain that, I'll let you read the posts. And eventually I'll fill in more background.

I've used "TWGR" in some of the posts - that stands for The Well-Grounded Rubyist, the best programming language book I've ever read and currently my bible.