Updated 30 September 20
Who knew software development could sound so simple? Nova's Chief Technology Officer, Andy Speakman, draws on his extensive experience working with early-stage tech businesses to share an insightful overview of software development for founders. 👇
Thanks everyone for joining me today on this engineering primer, which is as part of novas masterclass series and having watched a few of my predecessors, I know that the bar has been set very high. So I'm hoping not to let the side down here.
Rather than just go through a list of jargon in alphabetical order and try and explain them, I tried to put this presentation together in a way that sort of takes you on a journey. So I thought we'd start with writing a very simple program, running it on a local machine and then trying to build more complexity and then get it out into the world for use. So that's the sort of journey that these slides are going to go on.
So writing our first program. If we go right back to basics, a program is essentially; 'a series of instructions telling a computer what to do' and the languages, as you know, with spoken language, is a set of vocabulary and grammar rules. Because computers work in binary, they can only speak and process ones and zeros. So in order to write our program, we're going to need a computer. We're going to need a text editor to write our programs and we're going to choose our programming language.
So in the early days, languages were very close to the ones and zeros. You've probably seen old footage of men in lab coats feeding punch cards into very old computers, literally feeding them their instructions. But over time, these languages have evolved to become much more understandable to humans and also more portable. So you can write a program on one computer and it can be run on a variety of different computers.
So. I've basically selected the top 10 languages that are in use around the world. There are a number of ways that you can measure this, and some people use metrics like, well, which one is Googled the most, which one has the most training courses available, or there are programs that will run around the Internet and look for what technologies are underlying or underpinning the different websites that you use. I mean, I've taken an amalgam of two or three of these sort of top 10 charts and at least eight of them overlap so I've picked the one that suits us the most. And for those of you, that the 'hawk-eyed' amongst you will notice that while I've said top 10, there are nine logos. But fortunately for me, C and C++ in the top left hand corner are part of a family of languages with a lot of common history. So I've managed to stick those together, which makes my alignment nicer. And I think that's obviously very important anyway.
In terms of how we work at NOVA, you know, we use five of the top ten languages in use in the world right now. I think the reason I want to mention that is that it's to give you the founders in particular, some reassurance that we're not doing things that are necessarily bespoke or difficult to put away somewhere else. You know, I'm going to talk about departure readiness a little bit later on towards the end and I think it's important that you realise what we're using. Fit for purpose and well proven technologies that are things that you can take with you and are not going to be things that are immediately difficult to translate or for another engineering team to use, potentially.
So we've chosen a programming language and we're ready to start. However, we don't really want to write code unnecessarily, so reinvent the wheel, so to speak. So we should always look to take advantage of things that we know as libraries, frameworks and APIs.
I want to cover quickly what those are, these are probably phrases you've heard before. So with a bricks and mortar library, this is a collection of pre written books that the public can take benefit from. Similarly, a software library is a collection of pre-written functions that we can benefit from, usually by downloading them from free for free from the author, and we can use them in our own software. So, for example, we might download a library that makes it easy for us to format dates and times. Or we could download one to create a PDF file from a text file. Now we could obviously write these from scratch ourselves, but it would be better use of our engineering time to concentrate on writing code that makes our application special.
So a framework is similar, but it's a bit more wide reaching and it's a bit more powerful. In the picture of the house here, you can see that there's a basic structure of the house in place along with the foundations and probably some plumbing and drainage. So the builders can now concentrate on building the bits of the house that make the house unique; so the room layout, the materials and so on. In software terms, if you're building an e-commerce site, you know that you're going to need some login functionality. You're going to need probably payments, you're going to need products and you're going to need some basic navigation. So an e-commerce framework would provide you support for all of those elements so that your engineers can just build the bits that make your e-commerce site different. So that's kind of what a library and a framework is.
Now an API is incredibly important. It's a phrase you've no doubt heard many times. And so I put it onto a separate slide, because I think it's worth discussing a bit more detail. With libraries and frameworks we download those elements, those bits of code, and it becomes part of our own application. So they're kind of embedded into our own code. But that doesn't always work necessarily if you need to, for example, interact with third parties. So, for example, if you're doing payment transactions where the banks are involved or where there is an enormous amount of data which changes frequently. So as an example, thinking about address lookups. Now, if you can imagine a library for address lookups, it would be enormous, containing all of the post code and address information.
It would be updated every day so you'd be constantly downloading this library, it'd be enormous and you'd have to continually be rereleasing your software. So it just doesn't work that way.
However, we can still avoid reinventing the wheel in this case. We just need to find partners that offer software as a service or SaaS, as it's usually known. In this case, the service provided by the company is provided by their own servers, and we can access it via the Internet. And that part is all made possible by the API. You can think about the API as a website for other computers. So instead of it being a website that serves nice pages with text and logos and images and videos and so on, it provides services to other applications. So it's a website for other computers, really.
Now, I think probably a lot of you have a rudimentary grasp of what these are but why are these so important? Why is it on a separate slide? Well, I think that other than the fact that it helps us build our applications more quickly, it's your companies that can become SaaS providers. If you think about if I use an example of Menu Guru - Menu Guru provides filtered views of a restaurant's menu based on customer allergies.
Now, let's say Sunil, the founder, has an opportunity to partner with another company. If Menu Guru has an API, we can offer the filtering service to them too. So don't just think about using other people's services. Think about whether maybe your application or your product should offer its services to other companies too.
So now that we've chosen our language and we've written some source code and we need to keep them safe, after all, the code is the application and so it's a critical part of your business.
To do that, we commit our code to a source code repository, or you probably heard the phrase repine for short. So this means that the code is safe and it means that there is a single centralized copy which is version controlled. So if I can try and explain what I mean by that, this sort of prevents the famous 'the dog ate my homework' kind of excuse. You know, in our example at the moment, it's a single engineer that's trying to write a very basic application. But even then, he's got his code and if his computer crashes, then he doesn't have his code anymore. He may have backed it up, but he may not have more likely. So by putting the code into this repo or repository, we know we've got a very secure and very safe copy of the code somewhere.
The other important element about these repositories is that they offer version control, which means that every time we make a change to a file and we commit it back to the repository, it actually records the differences that have been made, but also who made those differences. So when you start to work with a team of engineers, you're actually able to track those changes over time. For example, if we release our software into the wild and we introduce a bug, the very easy course of action you can take is, "well, we don't think that bug existed before we last released the code. So what changes have we made in between?". You can go into the source code repository and you can say, show me all the changes that were made between last Thursday and today and it will quickly draw you up a list. You can see which files have been changed, what changes have been made to those files and even who made them. So you might go and track that person down and ask them what the change was made or how they made the change and why. So a critical piece of tooling that we would use and probably something you'll hear the guys refer to was a repo.
Here at Nova we use BitBucket as our repository, which is an online service. It's part of the confluence of the Atlassian family and the Atlassian family of software includes here and confluence, which are tools that most of you already know and probably have worked with in your time here at Nova.
So we've now selected a language, we've written some code, we've selected a framework, we've downloaded some libraries, and we're only on slide seven. So that's slightly worrying. We've also created some other assets that we might need, for example, video, audio or even format files. So in short, we could have at this stage dozens, hundreds or philologists, even thousands of different types of files which go to make up our overall application.
Now to make moving and running our application more manageable, we typically build this into a single file. This is now our application. What happens in this build process is really sort of out of the scope of this overview but if you've heard, for example, of something called 'compiling - which is the process where we take our source code and we translate it into the ones and zeros that computer understand -that's part of what happens here. But it's probably the thing to take away here is when we talk about building or if the team is talking about building, it's taking all of the files that we've written or the files that we've downloaded or the files we've created and compiling them into the application so it's easy to move around and to install elsewhere.
I should probably point out as well, knowing a lot of the faces that I remember have joined the the call, that this is a very general view of things and each language has its own peculiarities and if you are a bit of a software expert, you could probably pick the odd hole here and there and cite examples of where this isn't always the case. But I'm going to just stick to the general for now. And I don't want to delve into too many specialisms if I can avoid it.
So success! We've now written our very first application, which is a basic website which says 'Hello, world!' On it. I'll explain why the whole title of these slides has been 'Hello, World!'. When you're first learning a new language, then if you're attending a course, if you're learning from a book or if you're watching some online training, chapter one traditionally and almost always without exception, will teach you how to write a program that basically outputs the phrase Hello world - so that's why this introductory section is known as Hello World.
Now, there is a success here, but there is also a slight problem, and that is that at the moment this is a success, but only available to an audience of one. The program is running on the developer's machine and unless we're going to literally pass this laptop around from person to person, only the developer can see it.
That means that we're basically going to have to arrange some hosting and then deploy our application, so let's let's discover what these things mean. As we go out into the wild, we need to make our application available to the world and its dog, and this is the only slide where I've literally written down what I mean and I'm just going to read it to you. I think most times I've managed to find a diagram that sort of explains it, but I do apologize in advance but I'm literally going to read the slide to you in this case.
So, hosting companies effectively provide servers and other services which are available to the public and servers or simply computers that can serve requests from clients. So, for example, when you're browsing a website on your browser, you're sending requests to the website server and that is effectively serving responses to those requests. So if you visit the BBC website, for example, the service at the BBC gets dozens of requests for the icon, the logo at the top, the images, all of the text and it's basically serving that back to your browser that knows how to make it look nice for you to read. Deploying the application is simply the act of installing it on one of these publicly visible servers. So that's what deployment's all about. When engineers talk about building and deploying, it's about packaging up all of the stuff that creates the application and then literally installing it on one of these servers and therefore making it available to the audience. So the server is now the host of your application and that's where you get the phrase 'a hosting company' from.
Which hosting company or cloud hosting company should we choose? You know, there are a lot of offers out there, but if we have a quick look at the market, we can explain why Nova uses the hosting company we use. So there are three big hitters in the industry at the moment, the Zawadski, which is Amazon Web Services, which is Amazon's offering, quite obviously. Then we move to GCP, which is Google's cloud platform, an offering by Google, perhaps unsurprisingly. And then finally, in the Big Three, you've got Azure, which is Microsoft's offering in this area. They all offer very similar services. But we've chosen to go with Amazon and the reason is, if we look at the market share for these three organizations, you can see Amazon is really the front runner. And I would go as far as to say it's really the de facto standard when it comes to cloud hosting companies that provide services to software engineering teams.
You know, they started first. They were the first to really take this approach and they still lead the market. Azure is building a good following now. But it's still considerably behind AWS in terms of market share and Google, perhaps surprisingly to a lot of people, is still quite low in terms of the number of people, clients that are using it.
I've spoken a little bit about cloud hosting and skipped over too much detail there, but I just want to circle back and explain what's so special about cloud hosting and what is this cloud that people keep talking about. So to explain it, I want to take you back to a time. Shortly after I graduated, I mentioned that to Tubdale. That was one of the companies I worked for. And I wanted to make their application available to a large networking company with data centers around the world.
Now, a friend of mine whose birthday it is today, coincidentally so happy birthday, Roy, he and I drove one of these large cabinet sized racks full of computer equipment down the motorway to a data center in London where it was installed. So Tubdale had to invest in the computer hardware and the space at the data center. And the data center was expensive because it had to provide fire suppression, power and air conditioning, amongst other things.
On top of that, your team now really needs to have a specialist in it that understood how to manage these physical servers. So that's a really big barrier to entry for a lot of software companies in that time. Now, in all honesty, over the years, things did become slightly simpler and cheaper. But then in 2006, Amazon launched AWS. With AWS you don't need any physical servers. You simply log in, click a few buttons, enter a credit card, and you can magically create a server for you, which is ready for you to use within a few minutes.
Similarly, for another example, if you need the database, the same applies - so it's far simpler. You also need only pay for what you use and it's paid by the hour. So, for example, if you just need a small, less powerful server for a day, then that's what you'll pay for. If, on the other hand, you need 12 very large servers, you get a lot of traffic, lots of hungry applications running, and you need them running continuously, then that's what you're going to pay for, too.
So it's far more affordable. Finally, AWS is built to provide elasticity, which is a great word. But what does it mean? Well, it kind of means that you can grow and then shrink the number of servers you need on demand. So in the morning, when your website has less visitors, it might just keep one small server running. But as your number of visitors rises through to a peak number at tea time, it can automatically create another six servers to handle the traffic.
Then overnight it can drop back down to one. So that's where elasticity comes from. So in summary, I think, amongst other things, cloud hosting is simpler, affordable and it's elastic. So that's one of the reasons that we're using that cloud host. So we've now chosen to let me see if I can remember what we've done so far. We chose a programming language. We got some libraries, we picked a framework. We've probably plugged in a couple of APIs. We've got all our assets together. We've built our software. We found a hosting company and we've deployed to a server. So we now have success - by choosing the hosting company and deploying it our 'hello world!' application is now available to the wider world. As you can see, it's no longer on localhost. It's at our 'product name'.com.
We need to move beyond Hello, World Hello, World is a very, very simple single-page website, but the reality is for most of the founders on this whole, you're already way beyond that and I'm sure you have ambitions to grow even further.
There's a lot of variety between the different products that we've worked on over the years in terms of what they're composed of but this diagram shows one simple example. At the top you can see that there are four entry points or applications that the users use to access the system. In this case, there's an admin website and there's a customer web app, a tablet application and a mobile app. These will communicate over the Internet with a Web server and an API, which are backed with a database to store the data, obviously.
So each of these may have been written in a different language. So the combination of the technologies used to create all of these are known as the technology stack - and stack, quite literally, because typically in diagrams you would see starting at the front end, at the top, you traverse through the middleware and the back and through to the database. So there's a stack of technologies. So you'll often hear about the technology stack and you probably have an interest in what technology stack your product is using.
From a skills perspective, this stack is often broken up into front end and back end. In simple terms, if you consider the front end to be the front of house, i.e. It's what the customer can see. So if you think about a shop, for example, it's the window dressing and it's where all of the I guess the well-dressed and presentable people are interacting with customers on a day to day basis. The back end is the behind the scenes where the grunt work is done, where the data processing and a lot of the logic happens.
So you will hear developers described as frontline developers for backend developers. And it just means that they've got skills suited to the languages that sit in either of those areas. However, if you find engineers that are equally adept at both, then that's where you're going to find your famous 'full stack engineer'.
Unless you're going to be working with one super productive full stack developer who builds everything for you because you're going to grow your application, you're going to need multiple components, you're likely going to need to grow a team - or you may already be working with a team. And as you know, in any industry, that can be a challenge. You've got to work with interpersonal dynamics. You've got to work with different strengths and weaknesses, all of the good stuff that teams bring with them.
And that's why, I guess process and structure can become quite important. We're not process or structure heavy at all and over, but you need a little bit of it to help those team dynamics work and to help teams work together in the best way possible. So I wanted to spend a little bit of time just discussing how these work at Nova.
So we'll start with process. I think Nova is definitely an agile organization and certainly that translates into the way that we develop software.
I'm sure many of you have heard agile used and maybe wonder what it means in practice. But to describe what it is I think it's a good way to compare it to one of its alternatives, which is called a waterfall. So this diagram compares agile Waterfall in quite a crude way, it's a bit busy, so I'll try and take you through it, pick out a few key points above the grade project timeline arrow. You can see what is considered to be a waterfall project.
This is broken down into five distinct stages. So you can see requirements, design, development, testing and then finally deployment. I think what's really important to consider here is that in a large project, each of these stages could take weeks or even months to actually complete - I think this is the critical element here. If you imagine, let's say, for example, each is taking a month. It's going to be three months before you actually see anything in the testing environment for you to look at.
And, you know, there are very few industries in which you can afford that amount of time to wait. I can think of examples where I was at The Hut group and we wanted to introduce auto correct on the search bar. I think Amazon has done it and as with a lot of good e-commerce companies, we have people scouring their website looking for ideas. And when this came out, it was like, OK, how do we do it?
And then a waterfall example, you might spend a week or two writing out requirements, getting people into a room and then documenting it to the nth degree. You might have a 20 page document explaining exactly what it would be, exactly how it would work, exactly how it would be presented, how it would deal with errors and so on. Then the software designers would look at this big document, decide how to sketch it out and where we would make code changes, how to make sure it performed quickly enough. And then that specification would be passed to a group of developers who now two months later start to write code over the period of the coding. Now, obviously, we're following a very strict Bible of instructions from the requirements document. So there's no real room to make changes. So even though in the two months already gone, the goalposts have probably changed five or six times in the e-commerce environment. We don't care. We've got our document. We're getting on with it. We're going to finish writing it if it kills us, and then we're going to get it released.
So this is really the crux of the difference between waterfall and agile waterfall is a relatively rigid and time consuming sort of way of developing software. And if we compare that down below the line into the Agile project, you see these Loopt kind of approaches. Feedback loops is the way I look at it. So instead of doing all of those things as one enormous amount of work followed by another, we do all of those activities, but we do them much, much higher rate for a very much smaller part of the overall solution.
So we break our project up into bite sized things and we deliver them quickly, more quickly. You know, as you can see above the line, again, there's only one big deliverable at the end of that, Goldstar, the one big outcome. And if you compare that below the line, you can see there are three deliverables. So we've broken our project down. We've designed it. We've talked about requirements. We've implemented it. We've tested it and we've released it. But we've done it at a small scale to production so it's ready for its users to see.
So that's the core difference to me. To me, agile at its core is about feedback loops and course corrections. To me, that's what it boils down to. It's getting something out in front of an audience super quickly learning from that and then course correcting and making the next set of changes. And with Waterfall, obviously, you're not getting your feedback back for many weeks or potentially many months later, by which time you may have gone completely off course.
A much simpler but slightly less sort of technical version of that is here on this slide. This is something I've seen pinned up in at least three companies that I've worked with. It's kind of the ethos of agile, I suppose. I think in essence, this demonstrates agile quite nicely, because all the way through it, we're delivering something. It's not exactly what we expect the end thing to be. You know, the car doesn't come at step one it comes at step four. But in each case, you can see that the stakeholder is getting unhappy right to the end. Whereas in the agile approach at the bottom, you're seeing that the user is getting happier and happier all the way through because we're delivering iteratively, we're delivering each sprint, we're giving them something towards the end goal. We're not waiting till it's finished to deliver it. And I think that's that's a nice visualization and as I say, I've seen this up on the wall in at least three of the companies I've worked with. It's quite a nice sort of way to represent agile versus waterfall.
Now for fear of repeating myself, because I can't remember whether I told the boat story yet or not. So if anyone wants to shout. If I'm telling a story that was on the slide before, when I think about agile, trying to describe agile to people and I think about my time having steered a boat. My dad has a boat on a canal and I'm lucky enough to go and spend some time on it.
And I think anyone who stayed about knows that even though the boat is going in a straight line, you can't just leave the wheel and go make a coffee because any of you who've done that will know that within 20 seconds you're going to be into the back. So what's actually happening is when you've got your hand on the wheel, you just constantly making these small changes to the steer and you can feel the effect that the current is having or the passing boat is having and you just constantly moving the wheel - but the but the boat itself is moving in a straight line. But you're actually making these miniature course corrections all the time. You're getting feedback from the boat and you're just constantly making changes. And that, to me, is kind of a good description of what Agile is.
So that's process or at least a very high level view of agile is and why it's important. If we just quickly go on to structure the slightly oddly aligned diagram, which I'll explain. Here at Nova, I guess there are three tiers of engineering for you to consider.
At the top of that is the engineering manager tier and this is the most senior technical person that will be involved in a project. If you're early in your journey with NOVA, this is likely to be the first person that you're going to meet with here in Nova. Chris Alcock and I split the portfolio of projects. So we kind of look after different halves of the portfolio, but we do have a watching brief on all of those and as I said, will often be the first person that you're introduced to or the first person to sit in the very early meetings.
Once those meetings have got to the point where we know we're going to make a move and we've got some ideas for what we're going to build, we will nominate a project lead sort. So I've skipped the technical lead, but I'll explain that in a second. So the project lead is going to be the day to day person that sort of represents the engineering team and the delivery. So this is the person that makes the key technical decisions on the ground, you know, figures out and recommends what the team might need to include, you know, who should be involved and really works very closely with the product manager, and also the founder I would hope, to come up with ideas on what we're going to deliver and in what order and makes those key technical choices.
The reason I've got technical lead slightly off to one side is that the project lead is either a technical lead or is mentored by one. So we've got a tier of four or five very senior guys in the organization and they will either be directly involved in projects and if they're not directly involved, they will be mentoring the project lead. So you will always have either direct or indirect access to a very senior technical person that the project leader can escalate things to.
And then finally, at the bottom, you've got the engineering team, the engineers, the QA's and so on down at the bottom. Now, what are the questions that I got is what is the size of the team or how big should a team be? And I think when I first joined Nova, there was an answer to that question that was quite rigid. You know, we had engineering squads that would typically work on two or three projects. What we've moved towards is a more, to use that key word, again, agile approach where we really decide on the team size, depending on the challenge at hand and what the problem is that we're going to solve.
So we're much more comfortable now saying, you know what, let's start with a really small team. Let's maybe even just have a project lead working with a founder and a product manager to to create something, to prove something or to run some experiments.
And then once we've done that and the results of the experiments are in, let's augment that team with an additional engineer and then that might just grow over time. So rather than prescribe it and say it's three and a half people or five people, we tend to really try and be as agile as we can in terms of how many engineers work on a team and it really is going to depend on the size of the problem and potentially any time scales that we have in mind.
So I think I'm getting close to wrapping up. I know there's a lot of probably elements that are not covered here, maybe some terminology or jargon that unfortunately I've not had time for in this session but I will know at the end we can do some questions and answers and maybe you can talk about other topics I might do in future sessions. But this is a really key kind of sort of area I wanted to discuss with you, particularly as founders.
It's a topic that sort of I guess it's a phrase that we've started to use in Nova called departure readiness. And really this is about making sure that when the time comes that a founder wants to move on, potentially step away from the umbrella that no provides and go their own way. We need to be striving to do things that are going to make that as easy as possible.
So I just bring back some of the logos that I had on earlier slides. If you remember, Amazon Web Services is the biggest cloud hosting provider that you can find in the world, the five of the nine languages in that list, that there are five of the most popular in the world. And BitBucket is a hosted source code repository, which means it's available to you wherever you go. It's one of the top 10 hosted repos in the world, and it's underpinned by a technology called GIT, which is the most popular source code repository technology in the world.
So what I'm trying to, I think, illustrate here is the choices that we make here are very much designed to build in departure readiness. You know, we're picking quite well proven and widely used technologies. That means that come the day you do decide to move on, transition should be easy. You're going to be able to find engineers that can with these skills, you're going to be able to take control of an account with all of your services on it.
And if you hire someone more than likely, they're going to know AWS and they're going to know how that works. And similarly, you know, we can pass you the keys to the BitBucket repository and you've got access to all of your source code. In fact, we can do that now. Quite often, if there are founders that have a technical bent to them or they're interested, we'd be more than happy to invite you to your own repository and you can feel free to share or take a look at the code yourself.
So that's what departure readiness is about. And I'm hoping that that reassures you that the technology choices we make are for your benefit and they will make transition easier.
One final point on here, but I think it's worth making. Olivia has mentioned how useful it would be to some of the founders to have some kind of overview, personal over the course of the project, you know, your own projects and what technology you use. What is the technology stack? What decisions have we made? What components are there? All these things? And that's something we're working on right now. We're creating a template report that will pull together information from different sources and we can then present to you. So you've got an accurate guide to the technologies that are in use and how we're using those things and how we've created your products for you. So that's a work in progress. Now in engineering, we're pulling together and be delighted in the next few weeks to start distributing those to the founders.
Final thought, I suppose. I'm often asked what advice would you give when it comes to working with software teams? I think if we look at the clichéd view of software teams and software engineers, maybe they're loners who like to sit in their own, you know, in their own offices. They don't communicate well with other people. These are all clichés, obviously, and I'm sure you'll know by now that's not necessarily the case. But if we think of that cliche, we think of them as people that speak a different language and don't necessarily collaborate well.
And it's easy then to start communicating in JIRA tickets or via email. My biggest piece of advice here is to engage with the team, particularly as a founder, but equally to anyone within the overall organization or anyone who's on the call. You know, there's nothing better for a team to understand the context of why they're doing it and why they're creating this application for this product. I think the founders drive and the founders, the reason that they brought this to Nova, their passion for the project really translates into an excitement within the team.
The other thing to think about, I think, is that these engineers have built projects before and built products. So they've built websites to build mobile apps. They understand how to build an API and so they can bring things to the table. You know, I'm often talking to engineers about the fact that they should feel that they can maybe make a proposal or question whether we should do thing A or thing B, because they've got some experience that often founders and PM's don't have.
So I do want to sort of build a culture here where there's collaboration and engagement between the team, because I can absolutely guarantee that it is something that will lead to better products being delivered in a better way.
Have you got an idea for a tech business? Get in touch.