In Software Development
Is Software Development Difficult? - read the full article about software development 2021, Software Development and from Continuous Delivery on Qualified.One
Youtube Blogger
we may not be up there with rocket scientists and brain surgeons but i think that most people assume that you need to be smart to be a software developer its not an unreasonable assumption it certainly takes a certain kind of smarts to be good at these games but what kind and why if ai algorithms can translate instructions into code and they can then what is it about software development that makes it harder than this simple translation exercise and more valuable what makes being a software developer difficult [Music] hi im dave farley of continuous delivery welcome to my channel and if you havent already please do hit subscribe and if you enjoy the content here today hit like as well id like to begin by thanking our sponsors harness equal experts octopus and spec flow theyre helping us to grow our channel and their links are in the description below if youre interested in the topics that i cover on this channel then why not sign up for my mail list where i publish how-to guides talk about upcoming channel content and periodically offer big discounts on training courses and books again the link is in the description below the simple answer is simple to this problem of what makes software difficult its other people i think that theres some truth in that but today i want to look at this from a slightly more technical perspective ive said many times on this channel that i think that software development is about much more than only coding but lets start with coding and explore why it is that that isnt really at the root of our difficulties as i said in the introduction modern ai systems like gpt3 can take simple descriptions of a system and write code that works weve been surprised by that and while its certainly impressive i certainly thought wow the first time i saw the demos on reflection i dont think that its really very surprising gpt3 was trained to process natural language and impressive as it is as a technology people thought it normal that it could respond to inputs in natural language with outputs in natural language but was seriously impressed when it could respond to natural language inputs with programming language outputs why natural language is orders of magnitude more complex and more expressive than any programming language the grammar in programming languages is perfectly regular and consistent unlike any natural language im guessing that what really surprised us was that a computer could do something that we find difficult we find programming hard because of the need for precision when we are talking we are constantly improvising and certainly if you are anything like me often tripping over our words and accidentally miss speaking this is pretty fundamental to how our brains work the really clever part is in the listener despite me stumbling over words occasionally mispronouncing them or even saying the wrong words you will still understand me your remarkable fuzzy pattern matching brain holds some kind of model of the world which means that you can correct my mistakes on the fly as i make them with a fairly high probability that youll guess correctly what i really meant a computer is a lot less forgiving in fact its not forgiving at all unless were programming it to be there are many fewer right answers a typo in the wrong place for a computer and our space rocket explodes human beings are not good at this kind of precision but computers are so fascinating as the ability of gpt3 is to be able to translate an input like this into working code it seems to me that the really clever part is in interpreting the instructions in the first place not generating the code as the output in this case the instructions are actually pretty simple this description is really a kind of high level pseudo code description of the system these are the kind of requirements that i hate as a software developer this leaves no room for problem solving this is the only this is only a translation exercise so impressive as gpt3 is this seems like a relatively simple exercise for it to me this is realized in real projects by one of the real difficulties of software development we tend to assume that its some kind of mechanical production exercise when in reality its an exercise in learning and discovery in this example the learning and the discovery the hard part happened before someone came up with that pseudocode list of requirements we do that kind of thing in the real world all of the time if youve ever had a user story that said something like move the quantity edit box three pixels to the left you know exactly what i mean lets think about this from another perspective gpt3 is fundamentally picking the next character of output based on probabilities so if im speaking english and i write f o what comes next could be foggy forgotten floppish forgery any number of words what if i was writing this in java though it could still be a lot of different things if it were a variable name but if it was positioned in a way that we could tell that it was a language keyword something that we expect a compiler to determine all of the time then now its fairly obviously that this is the start of the word for its the start of a for loop so not only do we now know that the word is for but we also know the kinds of things that come next so we narrow our search space this is a much simpler problem to solve for a programming language than it is for a natural language so the simple translation of detailed steps into a functional program isnt what makes our job difficult the real difficulty is understanding what those steps might be we cant defer that to somebody else i think that development teams that try to blame poor performance on poor requirements are usually looking in both the wrong and the right place at the same time theyre wrong in that this is not somebody elses job to tell you step by step what code to write if you think that is your job then gpt3 can do a better cheaper version of it even if the company that you work for is mistakenly structured that way its still not what our job really is so if your product owners or managers are telling you where to put the buttons or which columns to add to your database theyre doing it wrong this is an attempt at programming by remote control the degree to which getting the requirements right is the right place to look is that figuring out what steps will take us in the direction we need to go and then figuring out if our direction was right is our job our job is solving problems with software not coding so we need to understand the problem that we need to solve and then try out ways to solve it i wonder how gpt would do if we asked it to improve how he interacted with our photo collection instead of giving it a list of steps or fix this bug where the system crashes if i upload two photos at once or refactor the code so its more readable and so easier to work on in future these represent a different kind of problem solving and one that at least for now only humans can do these things are vague unstructured and sometimes subjective actually i think that comparing how people think and work on code to how ais like gpt-3 thrink and work on code has somewhat limited value however smart they get a eyes are alien and work differently to us so how we optimize to do good things will be different to how ais optimize to do good things for example my bet is that each time you ask gpt3 to write some code it goes back to the start and rewrites everything from scratch its probably not yet great at data migration or refactoring humans dont work like that because were too slow we can solve much more difficult problems than gpt-3 but to do so we need to work incrementally and evolve our understanding and our solutions as we learn more ours is a discipline of learning and discovery what that means is that the incredibly difficult task before us a task that gpt-3 doesnt face is to sustain our ability to make changes as we continue to learn its easy to write throwaway code to solve simple problems but if the code isnt disposable or the problems arent simple at least to the degree that you need to revisit them occasionally then the ability to make sustainable incremental progress seems to me at the heart of our discipline and that is one of the things that makes it world-class difficult this episode isnt about ai its about what makes it difficult to be a software developer so lets leave gpt 3 and focus more on the difficulties the precision that our computers demand of us is challenging for us we have developed all sorts of tools and techniques to try and manage that programming languages and paradigms constrain the variability of our solutions in different ways to help us express ideas precisely in a form that is both understandable to other people and also executable by a computer immutability abstraction type systems functions modularity cohesion and so on are all tools to help us place limits on and exert control over the complexity at the heart of our systems and to allow us to build on what we have learned this complexity takes three different forms the first two are the accidental and essential complexity of our systems the essential complexity of a systems that we build is inherent to the problem that were solving if were writing a stock control system its essential complexity is all about stock and how to control it the accidental complexity is very different this is the complexity that we add to the systems that we build because our software runs on a computer the need to store things to disk because our ram isnt persistent through power off the need to render things on a display because we need to be able to see what were doing the need to take inputs from a mouse or trackpad or voice or all of these and many more are not really inherent to solving the problem before us but that doesnt mean that they arent important often the solution to these types of problems are pretty generic and weve come up with generic solutions to to many of them like file and folder abstractions for storage or windows buttons and list boxes for user interfaces but they are clearly a different class of thing when compared to the essential complexity of our systems in general if our aim is to sustain our ability to learn and to evolve our systems to fit with what weve learned then i think its a good general advice to try and keep some separation between the parts of the code focused on the accidental complexity and the parts focused on the essential complexity these things change at different rates the rules of stock control dont change as often as whatever it is that look we deem looks nice and modern in your user interface theres a third form of difficulty though and we dont often think about this explicitly i think its even more fundamental though that is the complexity inherent with dealing with information my aim here isnt to get too hippy and esoteric but information as an idea is a pretty fundamental concept there are some takes in modern physics that see everything only in terms of information at the most fundamental of fundamental levels ideas like concurrency are not about computers they are about information concurrency is the fact of two or more events or circumstances happening or existing at the same time if we have information in two or more places and those pieces of information can change they are concurrent if the changes to them are related in any way at all then the level of complexity has just potentially exploded this is a world-class quantum physics level kind of problem and to bring this down to earth for a moment we buy into this problem every time we spawn a new thread or decide to work on a branch in our code base the degree to which this matters is primarily about timing if i change my copy of some information and you are interested in it then youre going to get a very different result depending on whether you look at my copy before after or during the course of me changing it now lets imagine that we each have a copy of the same information each of us changes it independently of the other which change is correct maybe they both are this is an incredibly difficult problem at the level of software development process this is the problem that version control optimistic and pessimistic locking and continuous integration were invented to ameliorate at the level of code we use locks synchronization blocks immutability transactions and eventual consistency none of these things eliminate the problem though and often they come at a significant cost in terms of complexity and speed as soon as you have related concurrent information the difficulty explodes in some ways this is the root of that other arch nemesis of software developer coupling if i write code on my own and keep it in my own repository it can be as messy as i like i may be dumb to work that way because i will suffer when i come to change it so coupling matters to me a bit but its me that suffers if i get it wrong managing coupling in a larger system though is one of the crucial tools to sustain our ability to make progress this is because of concurrency if i have a service that you consume our services are coupled via the interface that weve agreed that they will communicate through including any information that we share through those interfaces one of the reasons that we might organize our work this way is to allow each of us to make progress a little more independently of the other concurrently but we are sharing information so now we must incur the costs of that coupling i either have to tell you each time i change my interface and we must coordinate our work together so that the combined information of your system and mine are coherent or i have to work to avoid concurrent change in the information that we share but ive been never changing it concurrency and coupling deeply affect our work in small ways and big ways managing concurrency and coupling is one of the secrets of being able to sustain our ability to make progress but its always difficult thank you very much for watching [Music] you
Continuous Delivery: Is Software Development Difficult? - Software Development