What is software engineering?

If you’re in the industry or looking to get into the industry you’ll find yourself a little confused as to some of the labels and roles given to computer programmers.

• Programmer
• Developer
• Analyst
• Architect
• Engineer

The first two terms are freely interchangeable, the dictionary term for a programmer is one who programs, the term for developer is one who develops. So is there a difference in the terms or the roles? No effectively these two terms are identical and is merely a preference in terminology. Even microsoft uses these terms interchangeably for their internal developers.

An analyst might not actually write code, in a lot of instances this will be a business analyst, someone who understands the business and also understands how to translate the business needs into terms that can be digested by a team of programmers or engineers. An analysts primary function is to write the specification or requirements documents required for a new project.

A software architect, is very much like the traditional architect. They are responsible for building the framework and designing the tools that the team will be using to develop the project. Software architects are senior level developers who have had years of experience in developing software and can quickly turn a functional specification into a framework which is maintainable and future proof. A software architect can also be responsible for mentoring the rest of the development team in how to use the framework.

An engineer is none of the above and could be any of the above. A software engineer is someone that subscribes to an engineering discipline.

Definition of software engineering

Software engineering is the “systematic approach to the analysis, design, assessment, implementation, test, maintenance and reengineering of software, that is, the application of engineering to software” ^[1]

Universities can’t teach you software engineering.

This is a crucial distinction to make because after 3-4 years of going to university to earn your degree and learning how to program computers in various languages. Designing relational database models, writing functional and requirements specifications, you are not actually qualified to start writing production quality code. Not without someone to mentor you. Or for that matter you are definitely not experienced enough to architect a solution for a company.

A short story

A software engineering graduate finishes university and starts developing software as an independent contractor, he’s smart and decides that his rates will be cheaper than everyone else’s as his costs are relatively low. Without knowing it he finds himself with a client, a small business that’s looking for a solution on the cheap. The first thing business asks for is a quote. So the graduate starts investigating with their database of choice and comes up with a quote.

The business receives the quote and then mulls it over for a week and then decides to go ahead. So the graduate gets out his tools and starts coding away. After a week he has a prototype up and working which is mostly functional to let the client see how it looks. The client reviews the prototype and says “great, I’ll take it straight away” and so the graduate grabs his software and trundles down to the business and installs the software on an old pc out the back. After he’s finished the business owner grabs the graduate and asks that he install the software on another machine, shrugging the graduate installs the software on a second pc, the graduate sticks around for a while and everyone is happy. The software is performing well, so he leaves feeling happy about the outcome.

The following day the graduate gets a phone call, the business owner is very angry and states that the software program doesn’t work and he’s losing business. The graduate is worried and so heads to the small business straight away, he walks in and investigates the software he installed on the machine out the back, everything is working fine. He then investigates the software on the other machine which he installed at the last minute. It’s working fine there. So what’s the problem?

After investigating and then discussing that he can’t find any issues with the software, the business owner reveals that he assumed that the software was supposed to be networked together. The graduate face palms, that wasn’t discussed as part of the original quote. And so the graduate goes away for another week after discussing how the software is supposed to work networked, and starts banging out code. One week later he finishes the networked version and runs some cursory tests and it works fine.

Eventually after much back and forth the graduate ends up with a functioning piece of software for the small business. The graduate was forced to under quote at every step of the way, the business lost revenue whenever the program encountered errors, eventually becoming angry and hiring another graduate to write another program.

So what went wrong?

• Was it the code being full of bugs?
• The business not telling the graduate up front about all of the requirements?
• Was it the quoting process?
• Lack of experience?

All of the above.

Software engineering could have saved the graduate from making all of those mistakes.

Software engineering is a relatively new discipline in comparison with say civil engineering. As a species we have been building bridges for thousands upon thousands of years. We’ve been building software for less than a hundred years, so it’s a zygote in comparison to a fully grown adult.

Software engineering is the discipline of how to write code which is testable (Unit Testable), is loosely coupled (Dependency Injection), well designed (Domain Driven Design), and quick to implement new features (Agile).

Even for professionals, we don’t get it right every time. We like to take shortcuts to save time and save the customer money, but we end up paying for those mistakes later on.

What a software engineer would have done

A software engineer works as a contractor for a firm, the software engineer is expensive to hire and takes a long time to come up with a solution. The engineer goes to the clients premises and takes his notebook and starts making notes and drawing diagrams. He then begins interviewing the client and asking questions in regards to how the business works, what are their current processes, what they are looking to achieve with the software they wish to write.

The engineer then draws up a functional specification including data flow diagrams and begins work on a throw away prototype. Using the throw away prototype he gathers the clients together and the clients discuss how the software should look and function. Several iterations of prototyping comes up with a solution the clients are happy with. Through the prototyping process the engineer uncovers that the software needs to be networked, it needs to handle GST, large sums of money, BPAY, Credit Cards, interface with an existing Point of Sale system and micro-transactions.

The software engineer finalises his functional specification and starts working on a quote based on those requirements. The quote is hedged as some of the estimates have a 50%-100% inaccuracy as to how long they will take to implement and test. The engineer hands the quote to the client and gets paid for the time spent on site.

The client is impressed with the Software Engineer as they have seen the prototype and can’t wait to use the final product. Even though they software engineer is far more expensive than the graduate. And spent 4 weeks onsite without writing any code, the client has the functional specification with screen shots and accurate descriptions of all the functionality that they wished for, and the process weeded out requirements that the clients never knew they needed.

The clients hire the Software Engineering firm to produce the final software product. The Engineer goes back to the firm and begins to work on the framework to the software program, after about a week he has the foundation produced which has 100% test coverage and will meet all of the requirements that the clients are looking for, except he needs some help filling in the finer details. The basic splashes of colour have been added to the canvas, the lines have all been drawn and inked, now all that is needed is for fine detail to be done.

The Engineer gathers his team of juniors and graduates, he also grabs a graphic designer, and 1-2 testers from Quality Assurance. The meeting consists of discussing the Functional Requirements. Who is the client, what is the software supposed to do. The graphic designer and the QA team ask a few questions and then leave the meeting. The graphic designer have been given the approved wire frames from the client and is starting to do the finalised UI Design.

The graduates and juniors continue the meeting and start discussing the framework. A couple of days pass and now the team knows how the framework is going to work and how the framework is going to meet the requirements.

The team then contacts the project owner (a representative of the client). The team will be doing development in a rapid 2 week sprint. Before the first sprint can begin they need to determine what functionality will be in the first sprint. The product after each sprint will be a small feature set of the final product but each feature will be fully tested and fleshed out. After discussing the list of requirements the team and client agrees on 12 or so items that should be worked on for the first sprint, 3 of which will take more than 2 weeks to perform and will not be in the testable product at the end of the sprint but however it will be worked on as a priority as it will be mandatory for the next sprint.

The tasks are divided up and given to individual team members, which they go off and start writing unit tests which describe the functions that they’re about to write. The next day the Engineer pulls the team aside and have a quick stand up meeting to discuss their progress. Tasks are reassigned and development continues. 2 days before the sprint is set to finish the Software Engineer calls a stop work on new functionality. He reviews all current features which have been completed and reviews their test coverage (Does this test accurately reflect the requirement, does the function fulfil the requirement), after the review the Software Engineer drops 2 features as they do not pass the initial inspection. The rest are merged into a test build.

The Quality Assurance team is then called to action, they review the test build against the requirements and determine any bugs that they find or any requirements that have not been met with the functionality that has just been implemented. 2 more features have been found to be defective, these features are dropped from the final product and then will be fixed in the next sprint.

The first sprint is over, 12 features are fully tested and go into the first sprint to be shown to the customer, 7 features are dropped from the sprint to be worked on for the second sprint. The client is given access to the first iteration of the software product, each feature that is shown is fully working and passed internal quality assurance.

The client is impressed. This is exactly how the program was described, and it only took 2 weeks to get a start on the software product. This process goes on for 2 more sprints and in the final sprint the client shows an Alpha version of the software product to employees to have a play with. After using the software they come up with new features that they want. The 2 features are easily catered for and are included in the final sprint, 1 of the new features was not easily catered for and required an additional quote or estimate. The quote was drawn up and an accurate estimate given – this is accurate NOW because the framework is fully fleshed out, the design is fully qualified and the team members know the ins and outs of the framework. The client weighs their options and decide not the go with the new feature that was quoted.

However after 4 sprints (2 months) the clients have seen evidence of fast progress, they have used the software program, the engineers have implemented new features very quickly (because their tests indicated to them what features were broken due to new functionality), and the software program is fully tested to work in a production environment.

The client was extremely happy with the final product even though the initial lead up to the final product took longer than expected, the final results were outstanding. The client made millions from the software product, even though they paid more for the product up front.

So why doesn’t everyone do this?

Lets face it, Software Engineering is Hard

One of the main reasons why people don’t follow software engineering principles or misapply them is due to the illusion that by NOT following those principles they will save themselves and their clients money.

I think we’ve all seen this happen, or even more recently heard of it on the news.

http://www.computerworld.com.au/article/351608/updated_qld_govt_blames_ibm_health_payroll_bungle/

Why do these failures occur? It’s difficult to answer and there is NO SILVER BULLET^[2] and no magical solution to stop projects from failing. The best we can do is to increase communication with the client. 20 years ago software engineering was all about fully gathering as many requirements as possible and then fully fleshing out a design and then telling a programmer what to do, like a Cog in a machine.

It didn’t work then, and it doesn’t work now

We’ve learned from those mistakes and have developed new practices to help avoid the problems we had with the Waterfall method of software production. Waterfall is expensive, and produces obsolete and ponderous software. A software product is only as good as its fully tested and qualified feature set.

Waterfall Method

Take the example above Company B is a Software Engineering firm also, but is using old software engineering techniques which have been proven to be expensive and slow (Waterfall Method).

The client decides to go with Company B. Using waterfall they gather all the requirements up front hire an analyst, and draw up a fully fleshed out diagram of the process of the entire software product including hiring designers to fully flesh out and draw images of the final product. It takes 6 months before they even hire a programmer to turn that fully fleshed out specification into code.

Sound Familiar?

Using the example above using new Software Engineering methodologies (Agile, SCRUM, Test Driven Development, Domain Driven Design, Design Patterns, Peer Reviewing, Quality Assurance). The entire process is fully done in 6 months.

When Waterfall is better than Agile

In the above instance is biased towards Agile for a singular type of solution (mainly web). Where most web applications are User Interface driven.

There are strong cases where the waterfall method is actually better than agile, and should be adopted instead. There are drawbacks to the agile software development methodology.

• Requires a good experienced team of developers
• Requires a client who is open to collaborating on the software project
• Requires a software product which has a User Interface for the client to use
• Requires more discipline to do documentation, agile does not mean No Documentation

Where waterfall is best used where an agile approach is simply Not Suitable. For instance writing COM Services, or server applications with no client facing UI.

When Agile should be used over Waterfall

Agile has its drawbacks, as does waterfall. Agile is best used with a product which has a strong UI focus, as it requires client interaction to help define the requirements.

The main reason it’s called agile is because in comparison to waterfall methodology. You can change requirements half way through implementing the software program and it will cost you less to implement. This is because using waterfall, if a requirement has changed, the entire process is scrapped and starts from square 1. Re doing the fully fleshed out specification document, and then changes to code becomes a nightmare because the framework has been written for the first version was designed to work with the first version and will require a complete re-write to work with the second version… it is very rigid.

The agile version doesn’t make assumptions nor holds anything in concrete, it is designed to be able to be changed between sprints, the requirements are not set in stone and so time is not wasted on fully fleshing out exactly how the software program is going to work in every minute detail. Which saves the client time and money on wasted effort.

You will notice however that even describing the agile method it is far more involved than when I described how the graduate (who used no process) performed the same job. Using no process in the end took the graduate longer than what he initially estimated, cost the client business, and cost the client more money than what they originally estimated.

Software Engineering is expensive.

Lets not be under any illusions here, software engineering is expensive, and old software engineering is even more expensive. However when its done right using what we’ve learned in the past 20 years it can also be done quickly and efficiently with guaranteed results.

¹ Laplante, Phillip (2007). What Every Engineer Should Know about Software Engineering. Boca Raton: CRC. ISBN 9780849372285.

² Fredrick P Brooks (1995). Mythical Man Month. Addison-Wesley Professional; Anniversary edition (August 12, 1995). ISBN 9780201835953.