Focusing software projects on delivery RSS 2.0
# Saturday, June 07, 2008

The slides and code for my presentation today at Microsoft is available here for download.  Please leave a comment if you were at the presentation so that I know where my hits are coming from.

Simon Munro

Saturday, June 07, 2008 6:39:15 AM (GMT Daylight Time, UTC+01:00)  #    Comments [0] -
Main
# Sunday, May 18, 2008

If you are considering a career as an IT Architect you need to pause for a moment and wonder if you want to label yourself as an 'Architect'.  While there is a general trend at the moment to clarify the term more carefully and formally, most of the people you run into will have there own preconception of what an architect is.  Lots of people run around IT giving themselves the architect title and, whether doing architecture or not, have given all aspiring architects a bad name.

So when positioning yourself as an architect, consider that the following types of architects have already set the perceptions of what an architect is.

The PowerPoint Architect

By far the most common type of architect is The PowerPoint Architect, these kinds of architects produce the best looking architectures on paper... I mean PowerPoint.  Great colours, no crossing lines and reasonably straightforward to implement... apparently.  The problem with PowerPoint architects is that they are so far removed from real implementation that architectures that they propose simply won't work.  The PowerPoint Architect is generally a consultant who, just before implementation is about to start, picks up their slides and moves to the next project - leaving everyone else to implement their pretty diagrams.  The PowerPoint Architect believes that software development is similar to doing animations in PowerPoint and infrastructure is about how to get your notebook connected to a data projector.

How to spot The PowerPoint Architect

The PowerPoint Architect gives him/herself away by scheduling presentations in meeting rooms and having so many slides that there is no time to go into the detail.  If the meeting has more business and project representatives than technical staff, it was probably organized by The PowerPoint Architect so that technical questions seem out of place and should be 'taken off-line'.  The PowerPoint Architect has also been known to use Visio.

The Matrix Architect

Named after 'The Architect' in the Matrix movie series, The Matrix Architect has been there so long that he/she doesn't know any other way.  Matrix Architects leaves no room for improvement, discussion or negotiation as the architecture was written by them eons ago and has worked fine, thank you very much.  Much like the scene in The Matrix Reloaded, The Matrix Architect has a personalised, well defended office and if you manage to get in, you simply have to leave by one of two doors - without getting a chance to explain yourself.

How to spot The Matrix Architect

The Matrix Architect normally has their own office and is well settled.  Technical books on CORBA, Betamax and other has-been technologies are proudly displayed on the shelves.  The Matrix Architect can also be spotted by their uncanny ability to work their way into meetings and throw curveball comments like "That's just like the SGML interface that we used on DECT and in my day..."

The Embedded Architect

The Embedded Architect creates architectures that are so huge and complex that removing them is similar to taking out your own liver.  Most of the time they do this for career stability or, if they come from an external organization are there to milk as much future profit out of projects as possible.

How to spot The Embedded Architect

The Embedded Architect is very difficult so spot during the embryonic stage when they are infecting the existing architecture and often once spotted it is too late.  The Embedded Architect often has a team of disciples that as a group understand the entire architecture, but individually know very little.  A requirement that new team members go on an induction course on the architecture is a sign that there may be an Embedded Architect somewhere within the organization.

The Hardware Vendor Architect

The Hardware Vendor Architect is actually a salesman with a reworked title.  The Hardware Architect's role is to point out the flaws in everyone else's architecture so that they can justify why the extra hardware expense is not their fault.  At Hardware Architect School, The Hardware Architect is trained in creating proprietary hardware platforms that create vendor lock-in.

How to spot The Hardware Vendor Architect

The Hardware Vendor Architect normally has a car full of pens, mouse mats and notepads emblazoned with some well-known brand which they use to assimilate the weak.  They also have huge expense accounts where they can take the entire data centre to lunch occasionally.  They are often heard saying things like 'You need a 24x7 99.999999% disaster recovery site'

The Auditor Architect

We are not sure of the origins of The Auditor Architect, because they are supposed to be auditing things, not creating architectures.  The Auditor Architect will always propose an architecture that uses spreadsheets for every possible system interface that requires each user to be a CA so that they can review the transactions before they are submitted (not to be confused with The Auditor Project Manager who uses spreadsheets for all documentation).  Since most organizations don't have that many CA's, The Auditor Architect represents a firm that can provide as many CA's as may be necessary.

How to spot The Auditor Architect

The Auditor Architect always wears a black suit, white shirt and an expensive tie in the latest fashionable colour and style.  The Auditor Architect will often go to great lengths to express that they are unbiased and just want to make sure that things are done correctly.  Most emails received from The Auditor Architect have spreadsheet attachments.

The Gartner Architect

The Gartner Architect has knows all the buzzwords and has all the supporting documentation.  They never actually put together a workable architecture but run ongoing workshops on the likelihood of the architecture looking a particular way at some point in the next six months to five years.  As soon as an architecture is established, The Gartner Architect uncovers some 'new research' that requires a suspension of the project while the architecture is re-evaluated.  Incidentally, sometimes The Gartner Architect is known as The Meta Architect.

How to spot The Gartner Architect

The Gartner Architect always does presentations with references to some research noted on every slide and the true test of The Gartner Architect is asking for the document that is being referred to - it won't materialize.  The Gartner Architect is often accompanied by a harem of PowerPoint Architects eager to get their hands on the material.  The Gartner Architect is often entertained by The Hardware Architect, provided that they represent products that are in 'The Magic Quadrant'.

The ERP Vendor Architect

True Architects for ERP systems do exist - but they hang out somewhere else, like in Germany, and not on your particular project.  There is no need for an architect on a system that if changed, self destructs within thirty seconds.  The ERP Vendor Architect is actually an implementation project assistant that is billed at a high rate.

How to spot The ERP Vendor Architect

The ERP Vendor Architect almost always has a branded leather folder of some really fun training conference that they went to in some exotic location with thousands of other ERP Vendor Architects.  A dead giveaway is if The ERP Vendor Architect and The Hardware Architect are exchanging corporate gift goodies - a sure sign that they are colluding do blame legacy systems for the poor performance.

The UML Architect

The UML Architect is not interested in any architecture that cannot be depicted using UML diagrams and spend a considerable amount of effort making sure that this happens.  The UML Architect lives in an object bubble and has no consideration that their intended audience never learned SmallTalk.

How to spot The UML Architect

The UML Architect is easy to spot from the documents that they produce.  All documents have a lot of stick-men, hang-men and and cartoon characters pointing at bubbles.  The UML Architect will always be able to describe the architecture by <<stereotyping>> it as something that you will understand.

The Beta Architect

The Beta Architect insists that the current version of whatever software you are using is going to be ridiculously out of date by the time the system goes live.  For that reason it is important that the development be done with the beta framework, operating system or development environment and not to worry, the product will be probably released before the system needs to go into production.

How to spot The Beta Architect

The Beta Architect normally wears a golf short with a large software vendors logo embroidered on the front and walks around with a conference bag suitably branded.  The Beta Architect normally comes from an external organization that has a partnership with a large vendor indicated by some metal, but always gold or platinum - bronze and silver partners are not worthy.

Simon Munro

 

Originally posted here

Sunday, May 18, 2008 4:56:44 PM (GMT Daylight Time, UTC+01:00)  #    Comments [1] -
Architecture | Archive

In the eighties IT had 'work study' and in the nineties methodology became fashionable.  If methodology was so important a few years ago, why is it less important or non-existent now?  Surely all the reasons for wanting to have the methodologies of the nineties still exist – it couldn't have just been a passing fad where everyone was sold snake oil, could it?

The short answer is that methodology was, and still remains an important part of delivering IT systems.  The need for methodology has not decreased; it is just that methodology is very unfashionable at the moment.  The familiar pattern of hype cycles in IT, such as data or centralized processing, go through cycles of being fashionable or not - methodology is in a slump and if it does come back into fashion it will look very different.

What is a methodology?

The idea of this post is not to define methodology and those definitions that do exist are a bit iffy. Basically methodology in an IT context is about the tools and techniques that we use as well as the processes around how we use them.  A methodology would, for example, guide how we take users' (data) requirements, turn them into a logical model and ultimately to a physical database.  All of the checking, feedback, versioning, rework and how to use the techniques themselves are rolled up into the methodology.

The Downfall of Formal Methodologies

There are many places to point a finger at what caused the lack of popularity of methodologies, but three stand out as having the greatest impact.

The Hyperlinked Generation

The last time I tried to get a bunch of users in a room for a series of formal sessions I spent most of the time running around the office, making phonecalls, re-scheduling and generally herding the group into a room – it would be easier to bath five cats at once.  The last time I distributed a specification document for comment nobody even read it properly.  Sound familiar?  If your methodology says that you need all of the users in the same room or that documents need to be formally reviewed then you are going to struggle to get your methodology implemented.

In today's business environment people are doing many things at once and don't really have the attention span that may be required by formal processes and techniques. Most people will have a few browser windows open, email, IM a media player and any number of things on their desk vying for attention – never mind mobile calls, text messages and other interruptions.  These days, a specification document is more of a media mashup from documents, emails and IM transcripts than an engineered document.

The methodologies of the nineties (and most heavyweight methodologies today) demanded attention that people are simply unable or unwilling to give.

UML

In the early nineties large organizations such as IBM and JD Edwards had huge teams working on methodologies and their related tools.  Although they kept an eye on object oriented techniques and considered them important, they were only a small part of the bigger picture.  As these heavyweight methodologies were maturing, three object guys shook hands and came up with The Unified modelling language – implying only one and that it was unified - modelling is a big part of any methodology.

The first, and loudest, criticism against UML was that it sucked as a methodology and while the proponents exclaimed "It's not a methodology, it's a notation!" the guys back at the office hastily assembled RUP (Rational Unified Process).  RUP was (is?) implemented as a heavyweight, document-centric, lethargic methodology that was rejected by developers because more people were required to maintain the documentation than actually do development.

UML was also rejected by business because the stick-man diagrams and 'useless cases' meant little to them – they preferred the good ol' flowcharts, data flow diagrams, ERD's and even IDEF diagrams.  After all, this so-called Unified methodology did not even have a way for business to represent process flow – business simply could not relate to the 'everything is an object' paradigm.

Waterfall is 'Evil'

In the mid-nineties the bulk of the methodologies being used were waterfall or BDUF (Big Design Up Front) oriented although they were not necessarily named as such. The movement from plan-driven to agile methodologies (Fowler) resulted in a development meme that the waterfall approaches were 'evil' and to be avoided at any cost. The reality is that few development teams actually took the time to understand the new methodologies and a long period of no usage of methodologies took place.

The software development teams had turned their backs on existing methods but were unable to phase in new ones – particularly with finding a way for organizations to painlessly move from document-centric to people-centric methods. I think that most organizations that are struggling to implement agile methodologies today did in fact have workable methodologies ten years ago that were abandoned with nothing to fill the vacuum in the interim.

Is Methodology Dead?

Methodology is certainly not dead – it has simply become unpopular and gone underground.  These are some of the underground groups.

The Silent Groups

There are a lot of people out there producing software and building big systems.  They use methodologies – they just don't call them methodologies and they don't talk about them.  They are not churning out daily builds with Ruby and have big waterfalls, long processes with lots of people that are doing something other than coding.  The reason that they keep quiet about it is because the market, competitors and even drinking buddies poke fun at them.  Fashion dictates that we should not wear comfortable shoes in public even if they are, well, comfortable.

Microsoft definitely uses methodologies but they don't talk about it much, it doesn't make marketing sense and they are not really in the business of selling development processes.  Imagine telling everyone how great your development processes are and then releasing a product like Vista late – everyone will be saying 'Microsoft development processes suck and they are not agile enough!'.  Okay, so in that particular example people are saying it anyway – but you should get the point.

The Germans, always known as being good engineers, definitely use methodologies and they don't talk about them much either.  SAP calls their methodology ASAP (AcceleratedSAP) and even use the word 'methodology' in their definition.

The Agilists

Some of the definitive works on Agile and Scrum made explicit use of the term 'methodology' to describe the approaches but they were written before methodology had the negative connotations that it is currently saddled with.  But very few agile users would acknowledge that they are using a methodology – it has too many negative connotations and is automatically associated with waterfall (which is evil).  Agile teams have approaches, principles and manifestos – not methods or methodologies.

I think that the problem with putting agile and methodologies in the same sentence is that agile methods give rise to multiple methodologies which are not formally described and documented for use by a particular development teams.  The agilists sell lots of books and seminars and hang out in private corners of the Internet, but seldom actually proclaim their use of formal methods and techniques – nor do the development teams understand that agile methods expect a team to choose its own process (Fowler).  Unfortunately this lack of formalism and secret-handshaking results in many development teams following some sort of Scrumbut approach ('We're doing scrum, but...').

Perhaps the authors and the leaders in the agile domain are trying hard to reinforce the correct use of methods but the users are not reading the books and articles – it is far easier to download a 'Scrum in 5 Minutes' document and implement it without even bothering with the detail.

The Architects

So if you are not part of 'The Silent Groups' or 'The Agilists' does that mean that you have a team of Cowboy Coders?  Many development teams produce the same sort of software as teams using a more recognisable (or even agile) methodology without officially following a method at all – but at the same time not making it up as they go along.  I believe that this is in fact the biggest group but how do they get it right?

The answer is that any successful development team does use methods and they configure those methods into their own proprietary or configured methodology – even if it is not recognised.  The distinction is that the methods are far more technical and the documentation is mostly in the most definitive requirement document that there is – the actual code.  Due to the low level technical nature of the methods the architects are generally the ones that are driving and implementing the methodology – mostly unknowingly to themselves and definitely unrecognised by business or even project managers.

Some examples are obvious – I, for example, use the built-in diagrams of SQL Server to represent, describe, document and even model my (MS-SQL) physical database.  There is no need for some external ERD tool that offers few benefits and is always out of sync.  Interfaces are another area where specific methods come into play – consider IDL and it's more modern counterpart WSDL – if software is going to interface correctly, across say a web service, then in order to define the interface correctly various processes have to be followed.  XML Schemas are another similar example where there is little space for cowboy programmers – your data will simply bounce back.

Good architects are aware that the technical choices that they make have a huge impact on how the system is developed.  An architect that takes a more object-centric approach (as opposed to data-centric; read DataSets) automatically has an environment where methods such as TDD (Test Driven Development) or even Domain Driven Design become attractive and value-add methods and techniques.

Even simple responsibility allocation can lead to different methods being used. An architect or technical lead may say particular developers work on the front-end, others on the back-end and so on. Such segregation requires that developers communicate, through agreed structures or interfaces, with one another. 

This concept of architects selecting the methods and directing the overall methodology is not lost on methodologists or product vendors. The increased awareness around DSL's (Domain Specific Languages) is evidence of the understanding that architecture and engineering types are moving away from loads of documents with stick-man drawings to something that is more useful to them. 

This is not lost on product vendors such as Microsoft who are trying to productize Software Factories in Visual Studio Team System. While Jack Greenfield from Microsoft is struggling to get buy-in to the Microsoft version of the Software Factory, the fact that the ideas are more aligned with software architects and engineers as implementers of the teams' methodology to me indicates a better chance of success than the MDA (Model Driven Architecture) nirvana - which is still too document centric and has a waterfall (evil) feeling.

The Future of Methodology

On a daily basis when working in corporate IT environments you get the feeling that there is no basis, no method behind the chaos in projects.  However, if you are prepared to stand back and analyse the processes at work you will see that the disciplines are often there and development do try and do things properly, as an engineering team would.  The methods that they use may not be bound up in a huge book or corporate 'Development Standards' file given to new recruits – but in many cases they do exist.

The challenge is to understand more clearly what we are doing and to embrace the changes and new methods that will no doubt be possible with new tools and influenced by inevitable technological advancement in our field.  The biggest problem is getting the non technical influencers, such as business and project management to understand that we are neither selling them the latest fad nor behaving like cowboys.

 

Simon Munro

 

Originally posted here

Sunday, May 18, 2008 4:56:16 PM (GMT Daylight Time, UTC+01:00)  #    Comments [0] -
Archive | Methodology

The tale of the Pied Piper refers to all of the towns' rats being led away, followed by the children after he wasn't paid. I see the same thing happening with capable IT architects - the analogy works even if you can't decide if architects behave like rats or children. While corporate IT departments are busy with other things, the real architects are slowly disappearing.

Definitions of the architectural role aside, let’s for a moment assume that an IT architect is a technical person with significant experience making a technical contribution to a project. Look around within your own organization and count how many people who are actively working as architects that are above thirty five, in their forties and their fifties – I don’t count very many. It seems strange that with IT systems being so darn hard to build successfully every time, that there is a lack of receding hairlines on our technical teams. This is not necessarily the case in other industries – most surgeons are considered to get better with age, right up until they are too shaky to hold a scalpel. My grandfather finally stopped working when he was in his seventies, spending every day on civil engineering sites – he had built virtually every major bridge and road in the city and was useful to have around.

Where the IT Architects are going

If as an architect you can coerce a multi million dollar system into existence, chances are you have a pretty broad range of skills. IT architects, in addition to technical knowledge, can project manage, lead teams, handle politics, understand business and so on – all of this in a fast-paced, highly stressful environment. It is pretty easy to handle a career change.

Some architects have moved out of IT completely – I know some who have become pretty good property developers, while telling me how great it is because building technology is fairly static. Some stay in corporate environments, even consulting, doing things such as project management – it is far easier to brand yourself as a senior project manager than an architect. Some move onto the supply side of IT, using their experience to sell products, services and resources to corporate buyers.

Most do less and less architectural work on a daily basis, lose touch with the current technological hype and feel that it is not worth the effort to re-skill again (intentional tautology).

Why the IT Architects are leaving

The reasons should really be handled by a more substantive survey, but there are some things that really annoy architects and make them throw up their hands and think "I'm going to become a project manager!" Here are some of my theories.

Fifteen years of experience overridden by technical developments in the last year

There is a perception in IT that the more you know about the latest technology, the better you are. While this may be relevant for part of the project team, much of the work that needs to be done is not latest-framework specific. Imagine an architect with fifteen years of experience focusing briefly on non-technical aspects such as quality for three years – by the time he tries to become involved on the latest project he is considered out of the loop and drops a couple of rungs in terms of technical seniority. Architecture is a lot about abstractions and as much as developers will say how radically different v3 of the framework is, the abstractions still remain the same. At some point good architects say "This is the last framework/language/methodology/platform that I am going to learn if we buy into the 'next big thing' I'm out of here!"

Seen this movie before

Do you ever get the feeling that in IT we are continuously re-solving problems that were solved a long time ago? I'm not even talking about that latest language, framework or other technologies that could arguably be evolving. I'm talking about things like handling temporal data, long running transactions, concurrency and contention and so on. Often architects find themselves mentoring, arguing or persuading others that if they choose a particular approach they will run into well known problems that may have been solved thirty years ago. Obviously this is the architects role – imparting their knowledge and experience, but sometimes they are flat-out ignored by people who think they know better because the technology has changed. In these cases I often start my counter with "I've seen this movie before and it has an ending like Titanic" – by now, people on my team know that it is a prompt to pay attention to avoid pain.

He doesn't code

Amongst architects the discussion as to whether or not an architect codes is unnecessary. The problem is that the technical members on a project feel that the architect should be coding and their reasoning is sometimes valid. A good architect is stuck in the middle – too technical to be valued for business knowledge and not a good enough coder to earn street cred from developers. Even coding architects find it difficult to keep up because they are not coding all day every day.

Salesperson Competition

Pre-sales techies are the bane of virtually every architect – energetic semi-technical salespeople that have all the latest brochures, white-papers, presentations and anything else to get business excited. Sometimes by the time the architect becomes involved, business is so sold on the particular product that the train simply cannot be stopped. Instead of inviting architects soon enough, and face it they don't get as excited and can ruin a 'fun' demo, business holds back on architectural input until they say "We have bought this product and it is a strategic fit – please get it integrated with all the other products that we have bought"

False Credentials

I was involved in a project with an awesome architecture assembled and delivered by a brilliant architect. When the final testing was going on external auditors rocked up to do an assessment – fair enough, it was a bank after all. They conducted a few days' worth of interviews and inspections and in their auditors report proudly announced that the architecture was bad (it should have been in a cube and not a relational database). How on earth would an auditor know, where are his qualifications, what is the basis for his argument, can we see the report, when can we have a meeting? Nothing… no answers… nada! "The auditors have spoken and it is so recorded, entered and agreed." Architects, without recognised qualifications and the rest of business not really understanding what they do are always at the whim of roles that traditionally have more clout and credibility.

Make him a manager

Sometimes, the best way to keep experienced architects around is to give them an office and a larger credenza. That way they can justify the cost and still have someone who knows how all their systems hang together. The problem is that then as a technical manager the architect does less and less architectural work and does administration, putting out fires and ultimately more golf. The result is an architect that is totally out of touch with what he or she really enjoys doing.

How to stop the Architectural Pied Piper

I don't consider myself one of those people-people and can't really provide input that is of a general nature – you can go and speak to your own people-people about that.

The first thing to decide is if you want the people with experience around – sometimes you may want to cut out the dead wood so that you can finally replace the token ring network. This can often be the case with architects that are simply looking backwards instead of forwards – the kind who look back fondly on SmallTalk and print out their emails.

Also you need to establish why you want to retain the experience – it could be that you need someone who knows all the history of the legacy systems so that they can be used as a reference guide for all the hacks and workarounds that have been put in over the years. That kind of person may be quite happy being a manager of a maintenance team and does not want any architectural responsibilities.

If you want to keep good, experienced architects, you need to create an environment where you can realise their benefit. A good place to start is to define the architectural role carefully – not simply employee specific roles and responsibilities. What is needed is a clear understanding of what architects do and how they fit into the IT department and it needs to be communicated more broadly. Although good architects are able to explain their functions to whoever they interface with, it would be a great help if all participants, from developers to business are clearer on the functions. Understanding functions within an organization is usually general knowledge – most people know the difference between say a sales manager and a financial controller – why should it be any different with IT.

For other ideas, look at the reasons for leaving above and find ways to counter them, such as:

  • Insist that architects have relevant current knowledge of implementation issues, but don't insist that they code it up themselves or patch it into the network.
  • Bounce proposed approaches off architects and ask for their input. Most good architects will be able to provide useful pointers, anecdotes and long stories of things that you have not considered yet.
  • Involve your architects throughout the entire procurement process. If they rough up the sales person a little – let them. The worst that can happen is that they uncover some cracks. If the product is worthwhile and the salesperson any good they will be back for another round – hopefully having addressed some of the concerns.
  • Back up your architects and present them as key participants in decision making – if they disagree, ask them to present their reasons formally, logically, concisely and clearly. Good architects revel in such an approach.

In case you were wondering, I'm not some washed-up techie who is going through a mid-life crisis. I have been in software development for about fifteen years and been doing architecture for at least eight. When I need to, I can out-code and out-deliver most developers on my team on the latest technology. I'm not leaving, although I if my current framework/platform (.NET/MS) becomes marginalised, I probably won't learn a new one in as much detail.

To the architects out there – see what you need to do to stay in the game. To business – keep your great architects so that systems can be delivered.

Simon Munro

Originally posted here

Update

It is interesting to read this post again about 18 months after it was first written.  I can see what I was frustrated with at the time and have, in the intervening months, tended more towards the detail of technology and solving of day-to-day coding problems.  I think that although there is a need for architects, the positioning is still a problem.

Sunday, May 18, 2008 4:55:19 PM (GMT Daylight Time, UTC+01:00)  #    Comments [0] -
Architecture | Archive

In interviews I need to know how well the candidate knows databases and start off by going back to basics.  After establishing that the candidate knows that the 'R' in RDBMS stands for 'Relational', I follow it up by asking for a definition of 'Relation'.  I will accept answers along the lines of "... a set of tuples" or at a push, "... a table" (A table is a lay term for a relation, particularly when used in the context of SQL databases).  However, most times I receive an answer that starts something like "A relation is when you have another table with a foreign key...", at which point I ask "Are you talking about a relation or a relationship?", which is followed by stunned looks or muttered agreement.  I have interviewed hundreds of people over the years and only one or two haven't used a 'foreign key' based answer.

For some reason the concept of foreign keys in SQL databases have been confused with the relation.  I think it comes from the colloquial use of statements such as "Relate employee and employer on EmpNo" - the 'relate' really refers to a relationship but since the speaker may remember a single semester course in relational theory (apparently making him an expert) recalls that it sounds a lot like the 'relational' in relational database and thus the confusion starts.  Maybe it was the result of Bill Clinton's infamous statements where, I can imagine, during a brief period when mathematics was more practical than abstract, a group of exhausted mathematicians decreed "After much experimentation we can safely say that 'sexual relations' are not part of the relational model and probably refer more to the physical relationship between two somewhat consenting adults"

I'm no mathematician sitting in some old ivy covered building at an academic institution - I deliver systems based on SQL databases and (ADO) DataSets every day and am a big fan of the technologies that I use.  However, having had a background in more formal methodologies and working with great, intelligent people over the years - I have tried to understand the theoretical basis for what I use and a smattering of relational theory fits into that understanding - I think that some knowledge of relational theory is important to build systems on top of SQL Server.

With that as a background, hopefully you will understand why, when using ADO DataSets and Visual Studio, I cringe every time that I have to refer to the ADO version of 'relation'.  Even typing that last sentence makes my hair stand up.

DataSets don't seem to officially claim to have anything to do with relational theory and with that as a disclaimer they don't do anything wrong.  Just as SQL Server or Oracle don't make any claims about fully supporting the relational model for fear of upset computer scientists and mathematicians spray-painting integrals all over the Microsoft campus.  However, since DataSets talk to SQL Databases which in turn are (sort of) based on the relational model - the distance between DataSets and relational theory is not that far - the inference that the DataSet relation has something to do with the relational model's relation does exist.   The only 'official' association that I have found is in MSDN (Datasets in Visual Studio Overview) which states "The structure of a DataSet is similar to that of a relational database; it exposes a hierarchical object model of tables, rows, columns, constraints, and relationships." - Which should be enough to upset a few theorists.

The ADO Relation is simply incorrectly named and propagates the confusion that people have about thinking about relations and relationship interchangeably.  The relation is a fundamental construct in relational theory and has formal relational algebra, relational calculus and other mathematics behind it.  The ADO relation is simply a badly named foreign key.  It is a fundamental error and is like calling the steering wheel of a car a chassis.

Not only does Microsoft abuse the term, they also use it inconsistently.  Visual Studio developers that have used the DataSet designer are used to the term 'Relation' as it is in the IDE (right click | Add | Relation) and I have myself said to developers things like "Create a relation (cringe) between those two DataTables".  However, the actual class that is being created is a DataRelation class.  Thankfully, the class is not called a relation and DataRelation is sufficiently devoid of significant meaning (like calling the aforementioned steering wheel a DirectionChassis) that one can get away with it.  But why then does Visual Studio call it a relation, and why couldn't they just have used 'relationship' in Visual Studio? 

Unfortunately it doesn't stop at Visual Studio.  The DataSet class has a property DataSet.Relations which returns, not a collection of DataTables as someone with some understanding of the relational model would expect, but rather a collection of DataRelations (er... foreign keys?).  DataSet.Relations.Add() does not add a new table, it adds a new DataRelation(ship).

To add to the confusion, the msdata schema definition (xmlns:msdata) uses the term 'Relationship' (msdata:Relationship annotation) or the xs:keyref annotation for foreign keys.  Generating DataSet Relations from XML Schema in MSDN recommends this after creating more confusion by stating "In a DataSet, you form an association between two or more columns by creating a parent-child relation." (aaargh! cringe! What on earth is a 'parent-child relation'?)

I think that Microsoft's abuse of the term 'relation' is shocking and breaks thirty-odd years of sound theory as well as breaking a few object orientation rules along the way by not naming things clearly and consistently.  I don't think that it is going to be an easy problem to fix, but the easiest change would be to rename 'Relation' to 'Relationship' in Visual Studio - most developers wouldn't even notice.  The DataRelation class is more core to the framework but is thankfully so badly named that it has no meaning anyway so it could almost be left alone.  I don't know how easy it will be to change the Relations property on the DataSet which is horribly orphaned and should have at least be named DataRelations from the beginning.

The number of people reading this post can never be as much as the number of developers going - Right click | Add | Relation - perpetuating the confusion.  I don't propose a Relational Vigilante Group that spray paints notations of set operators across the Microsoft campus and I imagine (sadly) that the generation that developed and believed in relational theory - the basis for nearly all our business systems - are literally a dying breed.  The father of relational theory EF Codd died in 2003 at the age of 79 and CJ Date, a driver of the relational model, is probably feeling a bit old and won't be able to compete with millions of young, energetic Visual Studio developers churning out thousands of blog pages every day.

Caveat Lector is is a Latin phrase meaning "Let the reader beware".  All that I ask is that as a .NET developer that you be aware of the terms 'Relation', 'Relationship' and their respective meanings in relational theory, ADO, visual studio and I suppose, the social sciences.

Simon Munro

Originally posted here

Sunday, May 18, 2008 4:54:44 PM (GMT Daylight Time, UTC+01:00)  #    Comments [0] -
Archive | .net

I had a torrid time with the hosting of my existing blog and had no access to the final content for far too long.  I also decided earlier on in the year to re-brand around deliveryfocus.net and didn't really find that delphi.co.za fell off the radar.

I don't want to lose my content and have decided to repost some of my posts here for reference.  Although reposting content is frowned upon, I feel that it shouldn't upset too many people.  There is very little overlap between audiences and I will repost the longer more essay-styled posts that have a longer shelf life.

Apologies to those that may have read the coming posts before.

Simon Munro

Sunday, May 18, 2008 9:04:29 AM (GMT Daylight Time, UTC+01:00)  #    Comments [0] -
Main
# Thursday, January 24, 2008

From the side I watched a project crash - mainly because the project manager allowed (encouraged?) key people to leave, replacing them with similarly skilled developers who had no experience on the actual project.  A previously highly productive development team started to languish and falter - missing deadlines an killing morale; a downward spiral from which the project could not recover.  In software development this is a fairly common scenario - particularly when developers start becoming cheeky and demanding (although we would never do that).

Some professions and industries are more able to handle the replacement of one person with another with little or no impact to the job being performed.  Semi-skilled workers and tradespeople are good examples, but even highly skilled mature professions can handle it with relative ease - the medical industry even uses the term locum to refer to 'a person who temporarily fulfills the duties of another'.  Of course, from an HR or pointy haired management perspective it would be great to be able to treat developers as resources that can simply be interchanged as if they were perfectly fitting parts of a well oiled machine.  Managers tend to think 'Hey, this guy knows C# so we can use him to replace that other C# person' - how many times have you heard that?.

As much as software developers think that they are special, the inability to interchange people is not unique to software development.  With any job there is a period of lower productivity when someone is replaced.  There are many reasons that can be applied to many jobs, from simple things like not knowing where the photocopier is to soft issues of being new and untrusted within a team to more complex reasons such as new skills that need to be acquired and the ramp-up time to understand new processes, terminology and organizational structures.  I would imagine that any manager would be familiar with these sorts of issues and most managers would concede that replacing anybody, developer or not carries a certain overhead.

However, development is a fickle art and for the unaware managers there are some issues specific to software development that need to be considered.

The developer may not know much that is relevant

The frameworks that we work with these days are huge, broad and continuously evolving either directly from the vendor or from third party suppliers.  It is virtually impossible for a developer to know, at a detailed enough level to be productive, the entire framework or technology stack.  So when a developer makes it through the interview process as, say, a senior Java developer, you need to consider what that means in terms of your particular project.

Given a set of 'things' within the framework, an a subset of 'things' that the project needs and the 'things' that a particular individual knows...

DevSkills1

If you are very lucky the developer knowledge will overlap quite nicely with your particular project needs...

DevSkills2

but unfortunately it is likely that the developer will be a touch out of sync with your project...

DevSkills3

and the effort to move the blue circle to cover the yellow circle is costly, painful, time-consuming and often fruitless.

Developers aren't as interested in the clients' business as you may think

Developers are continuously pulled in to meetings with business - sometimes to take a beating for issues and sometimes because people think that if the developer is in the session then properly documented requirements are unnecessary.  In these meetings a normally quiet developer will say something like "Well actually, you only need to re-order pink widgets if after they have been dispatched to the DC as they are the only ones not yet at the outlet otherwise you will have to cancel the order and issue an IBT" - a stunned audience instantly appreciates the developers insight into the business issues and the deep understanding that the developer has of their business.  But it is only perception - the reality is that at some point the developer had to cater for this undocumented exception(al) scenario in order to get something working.  The understanding that he/she has has come from experience rather than a fundamental desire to understand how to move various widgets around the world.

Developers, when they start out in their career are keen on understanding how business works and over time those that maintain such interest go the business analyst route or become assimilated into the SAP consultancy market.  Senior developers, although they have a lot of experience across various business processes, are more interested in how the code works rather than the business.  Very few top developers can have the same passion for business and development.

Yet, because of the occasional insights and business comments that are expressed by developers from time to time, management is under the impression that the developers are very interested and keen on the real business issues - a fair deduction from someone who doesn't understand the code at all.  Developers may monitor hundreds of technical blogs daily, but none relating to the actual business.  Developers buy books on the latest technology, but never on, say, the latest thinking around operational risk management.  Developers know the business issues because they have to code them up and understand how it all fits together - it almost soaks in over time rather than being studied or aggressively pursued.

When hot-swapping developers, this knowledge about the business simply walks out the door - much the same problem that one would have with any person in the organization with experience.  The difference with developers is that it takes time, effort and inclination to rebuild the business knowledge.  Think about the last developer that joined your project, maybe they read the requirements document briefly until their network login was set up, but as soon as they could logon and open the project they delved into the code, permanently shelving the requirements document as they started working on the system.  They in turn would implement rules in code, not comment them and never reverse update the requirements document - something that their replacement won't even notice.

Googling the code

When developers need to find out how to do something, it is easier to Google for an answer than it is to ask the person sitting at the desk next to you and we have not yet progressed to a stage where the code for a system, that is already easily accessible to all, can be Googled for solutions.  Yet at the same time, the chances are pretty high that the problem trying to be solved has been implemented already elsewhere in the project code, particularly if a strong architecture is used with patterns that everybody follows.

Over time, after writing his own code, debugging others' code, figuring out who to blame when the build is broken and so on, the developer starts to gradually build his own internal index of where to find things.  Solutions then become simpler, such as - "How should I authenticate against the report server? Go look at the authentication for the web service which is in the WebSvcController module.  Ah, good idea!".  Sometimes a developer will do a find on the project source code for a keyword that seems unrelated buts points her in the right direction.

When hot-swapping developers, that code index and search engine walks out the door, and the new developer has to re-index all over again - the common result is a few re-inventions of the wheel until the index becomes searchable.

Losing the argument

In the Infomet methodology the loss of an argument refers to the problem in business analysis where users may give their requirements and understand the first models produced, but as the design progresses new terms and abstractions are added and they don't recognise the final design - they cannot argue the correctness of the final model.  In software development a similar process occurs where changes are made, code is refactored and layers are added to a point where it is barely recognisable and particularly difficult to understand if the developers weren't along for the entire journey.

Any feature evolves over time and it is the most critical ones that evolve the most because of their importance and usage.  Take any simple feature like pressing a button on a form and a new page appears - the first feature complete iteration may be quite simple and have little code;  then it is decided to add some authentication and security;  perhaps there is a need to add some threading to improve responsiveness;  a refactoring process decides that some code behind the button is useful and it is refactored into another library;  another similar feature comes along that could use the functionality if it was abstracted correctly.  What you land up with down the line is a secure, threaded, refactored, abstracted feature - and it seems perfectly logical to the existing developers, who can also debug, re-use and find their way around the feature anyway, so they don't notice the complexity.

A hot-swapped developer will walk onto a project, take one look at the code the code and think "This is too frikkin complex, I think I need to rewrite it"

Hot-swapping vs Replacing

The idea of this post is not to state that developers are so precious and replacing them is too difficult and detrimental to the project.  Developers can be replaced and indeed should be replaced - you may, for example, choose to replace the initial developers with developers better at maintenance as the system stabilises as part of your normal SDLC.  Replacement however starts with the premise that the right skills need to be found, formal handover needs to take place and the process of replacing an individual is carefully managed.

Additionally, project managers should endeavour to have little or no high dependency on a particular individual and having a 'High Bus Factor' is a bad idea on any project.  This post serves to point out some of the development specific issues with considering developers to be easily replaceable as long as they have X years of experience in technology Y on their resume.

I sincerely hope that more project managers begin to learn that the term locum simply does not apply in software development - yet.

Simon Munro

simon - at - deliveryfocus - dot - net

Thursday, January 24, 2008 12:04:30 PM (GMT Standard Time, UTC+00:00)  #    Comments [1] -
Career | Project Management
# Tuesday, January 15, 2008

I had a successful couple of years blogging on delphi.co.za, a domain that I have proudly owned for about fifteen years - but the legacy and branding of 'Delphi' does not fit with what I do and how I think.  Scott Hanselman points out that using a consistent url for your blog is important and somewhere I picked up the tips of avoiding excessive cross-posting and making sure that you don't move your blog around too much.  So I tried to think of a url and blog title that would reflect my views and also be somewhat ageless - a url that contains that latest fad is either going to become stale or the blog will move around a bit.

As with many things, the first idea that comes to mind is often the best - try as I might I could not conceive of something more appropriate than DeliveryFocus.net.  I subscribe to many feeds and consume interesting technical information from people that are very clever and experts in their field - as much as I would like to consider myself a technical expert in some things, most of the professional work that I do revolves around delivery of software.  It is strange that with all the noise that is out on the Internet, very little emphasis is placed on what most people pay us for, namely delivering the required system on time, on budget, at a high enough level of quality - and a few other quality attributes.

Many tools and processes are about delivery - in a roundabout way. Agile and TDD are about delivery, but the message sometimes gets lost in all the fanatical rhetoric.  Some developer cultural wars are less clear - the ORM vs Dataset never-ending argument is a good example - regardless of the technical merits (or Object Oriented bigotry) one way or another there seem to be few considerations as to which approach provides optimum delivery capability given a particular set of circumstances.  Statements that contain "<some technology> is evil" fail to contain references to successful projects where <some technology> was used to get a system out the door quickly, cheaply and easily where the company may have gained a significant timing advantage over their competitors in the market.

My intention is not to write posts that are only statements only focusing on what it takes to deliver, but hopefully the essence of the need for delivery will come through as a common thread.  Why?  Because the work that that I am contracted to do relates to delivering software - I am not an academic who has the mandate to figure out the most efficient algorithm, I am not a rocket guidance developer where defects mean that people die and I am not, unlike many blogs out there, working for a large multinational vendor telling everyone how everyone except you is using the latest technology that I am talking about.  Instead I work in an enviroment surrounded by developers, project managers, users, facilities people, finance, vendors and a whole host of other people and organizations and I am responsible for getting systems into production that are architectural sound, of a good enough level of quality and meet the requirements of whoever is paying.  It sounds a bit far-fetched, but that is actually what most of us are doing.

So, in order to deliver software you broadly need

  • A good idea of the needs of your customer/users
  • An architecturally sound platform, tools, patterns and approaches
  • A team of people that are going to help you to deliver
  • An interface for all other hangers-on, project managers, facilities and competitors

... and the above list encompasses a huge chunk of software development - leaving me a lot to write about.

Simon Munro

simon - at - deliveryfocus - dot - net

Tuesday, January 15, 2008 4:00:56 PM (GMT Standard Time, UTC+00:00)  #    Comments [0] -
Delivery
Navigation
Archive
<September 2016>
SunMonTueWedThuFriSat
28293031123
45678910
11121314151617
18192021222324
2526272829301
2345678
Blogroll
About the author/Disclaimer

Disclaimer
The opinions expressed herein are my own personal opinions and do not represent my employer's view in any way.

© Copyright 2016
Simon Munro
Sign In
Statistics
Total Posts: 12
This Year: 0
This Month: 0
This Week: 0
Comments: 2
Themes
Pick a theme:
All Content © 2016, Simon Munro
DasBlog theme 'Business' created by Christoph De Baene (delarou)