Sunday, July 2, 2017

Category Theory and Corporate Culture

A mediocre poet once wrote

    Imagine me at 
             my age looking for 
    a job. Strangely exciting.

A sudden end to a contract - less than a week's notice, probably because the manager I reported to suddenly resigned.

Like a kid whose parents are divorcing, I ask myself if it's my fault.

It's not, really.

I was hired to help automate an operations support system for a communications infrastructure provider. Working alongside a developer (DW) whose assignment was to develop "automations" using a commercial application I'll call Grit (not its real name), it became clear to me that Grit's promise of easy development (even by non-programmers) was absurd. The "development environment" lacked basic amenities like unit testing and source control. Operations support in an industry like this requires a realtime event-driven system, but both Grit and the surrounding software environment were heavily batch- and file-oriented and required human involvement and slow manual workflows.

My coworker was so unhappy with the tools he was tempted to walk. I suggested we come up with a plan for a true realtime system based on microservices. We brought this to our manager (SN) who, in spite of not having software development experience, quickly understood what we were driving at.

DW has had a lot of experience with operations support and with web infrastructure. He provided the expertise that allowed me to concentrate on applying what I had learned in years of object-oriented development and reading up on functional programming and category theory. Furthermore, he supplied a perspective on system and application monitoring that my test-driven approach to development really needed. And he set up our "disjunct" development environment.

Due to some cultural issues with security, our Windows laptops were so locked down that we could not change even the most trivial settings in our browsers, and all software could only be installed from an approved list after a long drawn-out request process. There was no way we could do modern software development in that environment.

Thanks to the loan of a server from a middle manager, we were able to set up an IDE (Intellij IDEA Community Version) and access it through SecureCrt and XServer.  (Since we would sometimes lose connection to the server, IDEA was a safer choice than Eclipse because files are always being saved in the background by default.)

We successfully implemented the first phase of the framework, based on a fractal view of systems as consisting of total functions, dependent types and hexagonal architecture. The second phase would involve organizing the functions around models-as-types.

We couldn't put our code into production without official approval for the (extremely popular and well-tested) open source libraries we were using. This was another sign of the cultural chasm: there was no provision for software development because, as we were actually told at one point, "this company doesn't do software development."

In subsequent posts, I'll spell out the organization of the framework as it evolved and how I was able to bring some of the mathematical power of category theory and functional programming to the practical tasks of implementing an evolutionary development process that allowed for the maximum leverage of the batch/file/cmdline-based legacy systems we would have to communicate with - and perhaps eventually disintermediate, applying a more efficient variant of the Strangler pattern.

For now, assessing the matter of responsibility for what appears to be the end of that framework and that team, I have to blame myself because my background in anthropology and linguistics made me the only person who might have had a chance of understanding the cultural mechanisms that underlie Conway's Law.

But I don't blame myself much, because our team was doing some pretty serious development work, and DW, SN and our rookie developer JS were reasonably clear on the concepts. The cultural problems were operating at a "higher" level. Yes, if I knew at the beginning what I knew now, I might have been able to do something about the cultural problem.

But hey - I'm not a $500-an-hour consultant with all the flashy creds.  Just a developer with too many years of experience.

Thursday, October 29, 2015

Drive By Dialectics - Part 2

(Not @tottinge)

Part 1 stirred up a bit of a fuckstorm,  which surprised me less than how quickly it became a reasonable conversation.

Nobody cited the post as a LackOfGodwin violation, probably because there's no such thing yet. Maybe if I continue along this historically material path, there will be one.

But to be a little less inflammatory (get the Red out!), here's an analogy that's pretty much structurally identical:


(Party [├ťber]Animal)

A small group that communicates well doesn't need a non-performing leader. The key term there is not "small" but "communicates well".  There are conductorless orchestras, some of which are not particularly small.

Like a priest, the traditional conductor is believed to channel the sacred intention of the composer.

My role model, Charles Mingus - photo by David Redfern

Mingus could be as hard on his bands as any OberMusikenF├╝hrer in Germany, but he was also a great bassist and composer.

Got distracted by Mingus. Where was I? Casting aspersions on Scrum? What's the point? Sure, I'm a downtrodden victim of False Scotsman Scrum, but I live in Chicago. Do I have to tell you about my mayor? Worse than even False Scotsman Scrum.

Wednesday, October 28, 2015

Drive-by Dialectics

Ron Quartel, AKA @agileAgitator, came up with a nice analogy:


Here's my take:

    Scrum:Agile::Communist party:SocialDemocratic parties

The Communists perfected the concept of democratic centralism: an obvious oxymoron/contradiction that functions like a "Mystery" in a classic religion: among other things, it provides cover for autocrats. This lays the basis for what post-Stalinist Soviet leaders called a "cult of personality".  Witness Stalin, North Korea's "Dear Leader" and to a lesser extent Hugo Chavez and the Castros.

(If you think this kind of authoritarianism is incompatible with free enterprise, consider the PRC and Singapore.)

Scrum has triumphed in the corporate world because it centralizes control and information chokepoints in a "Master".  This reproduces the structure of a traditional corporation and promotes Agilewashing.

Other methodologies that essentially promote acentric/holographic democracy and the autonomy of teams are a near-impossible sell in a command-and-control world.

I have worked on four teams doing Scrum (obviously not True Scotsman Scrum - a period of dictatorship of the proletariat has to precede True Communism - substitute analogous rationalization).  One of them was run by an actual manager.  The other three were run by Certified Scrum Masters (blessed from Moscow?) who by the way were all genuinely intelligent and nice guys.

There was zero encouragement for team members to think about the process or about anything other than the microsilos (stories) they worked in - solo.

I don't doubt that these situations were more efficient/productive than classic Waterfall/Cowboy regimes.  But that's not saying much.

Friday, October 23, 2015

Closed for Permission, Open for Forgiveness

I used to think that the pattern I'm about to advocate here was unforgivable. Now I'm not sure - a side effect of test-infection.

I'm assuming OO here along with SOLID and DRY, and using the word "mock" because it's shorter than "test double", but not excluding "fake".

The controversial assumption: that tests are first class citizens and are therefore not forbidden to have an impact on production code.  In mechanical and electronic engineering, products are built to run "self-tests" long after they leave the factory: these tests are an integral part of the product and must be accommodated. Jordi LaForge ran Level 3 diagnostics in the middle of critical missions.  Freakin' Level 3, for Pete's sake! No controversy there.  But in software?

Suppose you have an object A that uses an object B. You want to verify with a test that A uses B appropriately. (Actually, because you're a smart programmer, you test-drive A to do the right thing.) You inject B as a constructor argument for A (no setter, because you understand the value of immutability).

So C, which instantiates A (whether C is a factory or a POLO [Plain Old Language Object]) now has to know how to provision/acquire/instantiate a B.  Maybe C is a container or a Springy framework, in which case you figure this is just part of its job.  But even in that case there's a dependency, no matter how cheap it is to create (say, with attributes/annotations).

If B is not part of a strategy pattern - if there's no "production reason" for injecting it - you're only doing it for the test.

What I've started doing is implementing a test-only constructor on A that takes a B argument so that I can inject a mock B. The production constructor instantiates B.

Frameworks like Hibernate make you create a default constructor so they can use reflection to set what should otherwise be private members.  That's nasty. Then there are the "beans" that require setters for every instance member when you really wanted an immutable value object. That's obscene.

Just sayin' - the test-only constructor is a microevil because nothing else has to know about it or use it.  Its visibility can be minimal, as long as tests can get at it.  For example, in a Java project where test source resides in a package structure parallel to but completely separate from production source, the test-only constructor can have default (≅ package) visibility. Similarly: assembly visibility in C#.

I'm not losing sleep over it.

Tuesday, October 6, 2015

Mandala for Developers

From Dead C++ Scroll 4762A - #DevelopersAreNotFreakinEngineers

(Note misspelling in lower left quadrant.)
Cf. Field notes XK7-2015 and JL9-2015

Friday, September 11, 2015

Best Release Notes Ever

In case you can't read that:

Slack 1.1.4
Mac SSB 1.1.4 Release Notes
- New: when viewing completed downloads in the flexpane you can open the file in its default application by holding down the shift key and clicking, with great flourish, upon the desired location.
- Fixed: a vile bug that could corrupt cache files if your Mac shut down or restarted unexpectedly while Slack was running.
- Fixed: a bug that would show greater than 100% completion of file downloads. This was not a useful area for overachievement.
- Fixed: A bug causing an erroneous zipper issue, automatically decompressing gzip files after they were downloaded has been totally repaired.

- Fixed: a bug that could leave you staring at a blank white window after signing in to a team whose session expiration time was set to zero. We now process the login result using a slightly larger value of zero. If you experienced this zero or less than zero times, congratulations. If you experienced it more than that, apologies, there is zero chance of you experiencing it again.

Wednesday, August 5, 2015

Developers are not freakin' engineers - Part 2

(Part 1: Developers are not freakin' engineers)

#Agile2015, which I can only afford to follow from a distance, came up with what looks like a new meme: Agile is improv.  And there's more that supports my assertion here.

Having spent time on the improv scene, I think the comparison is apt.  I'm gonna play with the metaphor.

1. Waterfall developers are actors who are expected to learn their lines and deliver them regardless of what's going on onstage or in the audience.

2. Engineering:SW Development::Stagefighting:Acting

Stagefighting requires mad skills: it's a domain. But a great swordfighter/MMA fighter/etc. is not necessarily a great actor.

Some domains are math/physics/etc.-intensive. Developers who work in those domains need all those skills. But they aren't enough to make for good software.

3. Here are a couple of quotes from the conference:

"Agile software development is people-focused and flexible." - via @DanaPylayeva

"Software processes which separate out design and construction are fundamentally untenable, unlike building a bridge, building electronics, etc. [Fowler]

The manufacturing metaphor was broken from the very beginning. Assemply lines are optimized to produce many of the same exact thing. With custom software, you only ever build one." - via @GetzTweetz

People-focused, flexible, custom - these are aspects of software that are seriously distant from anything we associate with engineering.

4. Software itself is orders of magnitude more malleable than what engineers deal with.  We're essentially working with pure Categories, magical atoms of Platonic ideal forms.

5.  I've worked with engineers and physicists who fled their abysmal labor markets for development. They knew a lot about the domain, but next to nothing about software.  Don't get me started about the code I had to salvage....

Engineers are not freakin' developers.

Developers are not freakin' engineers.