THE BLUEPRINT: 📚 Learning

Little AI Lessons

Sean 🤓 — Mon, 01 Jan 2024 12:00:00 GMT

This is a series of little AI lessons that I published on LinkedIn over the last 100 days of 2023. A lesson was published each day, covering a different AI-related topic, and each lesson had 5 levels of difficulty from beginner to expert. The little AI lessons series was designed to help people deepen their understanding of a wide variety of AI-related topics and technologies.

Lesson 1: Artificial Intelligence

Artificial Intelligence, or AI, is like a computer brain that can learn, think, and make decisions. It's a way for machines to do things that usually need human intelligence, such as understanding speech or playing a game of chess. Alan Turing, a British mathematician, was one of the first people to talk about machines that can think, way back in the 1950s. This technology is important because it helps us solve complex problems faster and can even do things humans can't, like analysing huge amounts of data quickly. If you've ever talked to Siri or Alexa, those are examples of AI!

Federated Learning is a machine learning approach where a model is trained across multiple decentralized devices or servers holding local data copies. Developed by Google in 2016, this technique allows for better data privacy as data doesn't need to be shared or copied. Instead, all learning happens at the device or server level, and only the model updates get shared across the network. This is particularly important in our digital age, where data confidentiality and regulation have become major concerns.

An Introduction to Artificial Intelligence

Sean 🤓 — Sat, 02 Sep 2023 22:00:30 GMT

This video provides an introduction to artificial intelligence (AI) and generative AI. It covers:

A brief history of AI, key milestones, and the main branches of AI like machine learning and natural language processing.
An explanation of what generative AI is - AI algorithms that can generate new content like text, images, and video.
An overview of ChatGPT - the popular generative AI chatbot from OpenAI that took the world by storm when launched in late 2022.
How models like GPT-3 and ChatGPT work - by predicting the next word in a sentence based on the context. Bigger models with more parameters display more capabilities.
Details on GPT-4, OpenAI's latest model, which is over 1 trillion parameters. It has new features like web browsing, plugins, and a mixture of expert models working together.
Reasons why the hype around generative AI is real - major milestones like 100 billion parameters, an intuitive interface with ChatGPT, and advances by the open source community.
A look at the explosion of use cases across many industries and early real-world applications of generative AI.
A summary that this technology will lower barriers to entry for knowledge work, and ChatGPT is poised to become a major digital platform. But it is ultimately just applied statistics.

An Introduction to Artificial Intelligence: Part 2

Sean 🤓 — Thu, 03 Aug 2023 12:40:47 GMT

Hi, and welcome to the second post in our Classroom series that covers an Introduction to Artificial Intelligence.

Why now?

One of the big questions around generative AI is why is this happening now all of a sudden? Why has this becoming such a hot topic and why has this technology taken the world by storm recently? There are four main reasons for that:

01 - Scaling Laws

The first reason is something called scaling laws. This is a piece of research that OpenAI did back in January 2020.

The hypothesis is that as you scale the size of the models, i.e. the number of parameters, you would see an increase in the capabilities that the models had. So the race has really been on to be able to create bigger and bigger models that then start to display more and more capabilities.

02 - Data Availability

In order to scale models you need to have a lot of data available to be able to produce models of larger and larger sizes. It's only been recently that large amounts of data has been available. You can see in the table above the list of datasets that GPT-3 was trained on. It was trained on something called Common Crawl, which is effectively a lot of text from the wider Internet. There's also a data set called WebText2, they trained on a lot of books and Wikipedia as well. This totalled about 500 billion tokens that the model was trained on, which resulted in the final model size of 175 billion parameters.

03 - Compute Power

In order to train models on large datasets you need to have a lot of compute power. The hardware that these models are trained on are graphical processor units (GPUs). They're not the usual computer chips that run your laptops or your desktops, and their production has been relatively constrained over the past few years. This is partly because of covid and partly because it has been harder to produce more powerful chips as the technology has shrunk.

For a good overview of the issues surrounding the production of computer chips I highly recommend reading Chip War by Chris Miller.

So the whole industry has been relatively hardware constrained in training these models.

04 - Transformer Models

Lastly is the transformer model architecture that Google developed back in 2017. This really was a big step change in how you build and train these large language models. Without these transformer models we wouldn't have seen the large language models that we see today. So it took Google releasing that back in 2017 to really unleash the opportunity to scale the models in the way that we've seen them scale.

Emergence

When GPT-3 was released back in 2021, it was the first large language model to surpass 100 billion parameters, which is a really interesting threshold as you start to see a lot of emergent behaviour from these models.

I love the graphic above, from a Google blog, which shows all the different emergent capabilities that start to present themselves as you increase the number of parameters in a large language model.

You can see how much more competent the models get, but also the breadth of things that they're able to do as the number of parameters scales. Emergence is an interesting area and it's something that we see in nature. For example, when you see a flock of birds and murmurations, that is a form of emergence.

It is essentially complexity coming out of a lot of small, simple decisions. And that's the kind of thing that we're seeing in these large language models and the reason why they've become seemingly so useful overnight.

Hallucinations

With the scale that brings emergence, you also get hallucinations. Essentially, hallucinations are when a generative AI model gives an answer that isn’t based on anything in its training data. The challenge is that generative AI models present their hallucinations very confidently, which can be very misleading, so it’s an area that is getting a lot of scrutiny right now. Hallucinations are one of the major reasons there are a lot of disclaimers on using generative AI models and every user should be aware that they can sometimes get factually incorrect answers.

I actually think hallucinations are a really interesting area. Broadly speaking, the industry is looking at ways to minimise or eliminate hallucinations, but I like to think of them as a generative AI model’s imagination. I believe we should be researching them more thoroughly so we can understand them better and look at how we can harness them for more creativity. However, we absolutely need a mechanism to flag to users when a model is hallucinating so we’re being completely transparent.

What is ChatGPT?

ChatGPT was built by OpenAI, which is a company that is only 7 years old and has less than 400 employees, which is tiny in comparison to the large digital platforms that they coexist with now.

OpenAI was founded in 2015 as a non-profit organisation and their mission is to ensure that artificial general intelligence (AGI) —by which they mean highly autonomous systems that outperform humans at most economically valuable work—benefits all of humanity. The founders of OpenAI were Elon Musk (who is no longer involved), Sam Altman, who is their current CEO, Ilya Sutskever, Greg Brockman, Andrej Karpathy and Peter Thiel amongst others.

OpenAI transitioned to a for profit organisation in 2019 because of the cost of the hardware required to train all of their generative AI models - they just couldn’t sustain their costs as a non-profit. The main beneficiary of this change was Microsoft, who invested one billion dollars and invested an additional ten billion dollars in early 2023. This increased their stake in OpenAI to about 49%, making them the majority owner and is the reason why the two companies work so closely together.

OpenAI released their first Generative Pre-trained Transformer model (GPT-1) in 2018, along with a research paper that demonstrated how a Large Language Model (LLM) was able to acquire general knowledge through pre training on a large, diverse set of text. GPT-1 had 117 million parameters and it could take a query (prompt) of just over a thousand tokens.

GPT-2 was released about a year later, had 1.5 billion parameters and could take double the number of tokens in a query. That was then followed a year later again by GPT-3 which had 175 billion parameters and was trained on data up to October 2019.

GPT-3.5 was released in March 2022, which was a similar size to GPT-3 but could take double again the number of tokens in a query. GPT-3.5 was trained on data up to September 2021 and they did something called finetuning, which is adjusting the model so that it was well disposed towards chat. This was the model that ChatGPT was released on in November 2022.

When ChatGPT was launched, it was the fastest growing technology that humanity had ever created, reaching 100m users in only two months. This has since been surpassed by Meta’s release of Threads, but that built off the Instagram install base of c.2.4bn users so is a slightly unfair comparison. ChatGPT’s user base has since doubled since launch to 200m monthly users as of May 2023.

GPT-4

More recently, in March 2023, we had the release of GPT-4. Unlike previous GPT models, OpenAI hasn’t officially released many details on GPT-4. However, speculation and research by the community point to some interesting features.

What we do know so far:

The number of tokens that GPT-4 can take has been hugely increased to 32k, four times the size of GPT-3.5.
GPT-4 will be multimodal, which means it is able to take in more than just text and also able to output more than just text as well. For example, it accept images as inputs and generate captions, classifications, and analyses. It can also output code, tables, data visualisations and many other types of content.
OpenAI also added plugins to GPT-4. Plugins are tools designed specifically for generative AI and help ChatGPT access up-to-date information, run computations, or use third-party services. These tools have a wide variety of use cases. For example, you can ask ChatGPT to look up flight times, book you a table at a local restaurant or read a PDF document.
One of the plugins that OpenAI initially released (but has since temporarily disabled for reasons) was Browse with Bing that allowed ChatGPT to access open webpages.

What we think we know so far:

It's thought that GPT-4 is an order of magnitude bigger than GPT-3.5 with 1.76 trillion parameters.
It’s believed that the 1.76 trillion parameters aren’t one single model but multiple models that are all fine-tuned and trained slightly differently. This is called a Mixture of Experts model.
It’s likely that GPT-4 has at least one model fine-tuned for safety and another specifically for coding tasks. Exactly how the models combine and work together is currently unknown, but something that I’ve speculated about in a previous post.
This is a really interesting approach that OpenAI have taken with GPT-4 and points to a future where multiple models are working together to get to the best answer, which has been shown by research to improve results.

In summary, GPT-4 has moved the capabilities of generative AI on another step with the introduction of multimodality, plugins and the mixture of experts approach.

Performance

https://arxiv.org/pdf/2304.06364.pdf

GPT-4 has shown it self to be incredibly capable across a large variety of tests and is currently the top performing large language model. As you can see from the graphic above, GPT-4 can already match top human performance in SATs with many other knowledge domains showing significant improvements over GPT-3.5 (ChatGPT). Every week more and more research is being published that demonstrates GPT-4’s ability to match or even surpass human performance across a wide variety of disciplines. We’ve seen examples of GPT-4’s capabilities in creative thinking, humanities and social sciences, law and plenty of others.

Where next?

During the course of Sam Altman’s global tour advocating for increased regulation of generative AI models, he dropped a few hints on the future plans for GPT-4 and beyond:

OpenAI want to increase the speed of GPT-4 and decrease the costs. (currently GPT-3.5 is 4x faster and 10x cheaper than GPT-4).
OpenAI want to increase the query size ever further, potentially up to 1m tokens like their rival model Claude 2 from Anthropic.
OpenAI wants to continue to develop the plugin ecosystem and the multimodal capabilities of GPT-4.
OpenAI have also hinted at a ‘stateful’ version of the GPT-4 API that remembers conversation history.
Open AI have also stated that they believe the scaling laws for large language models still holds, so I expect GPT-5 to continue their trend of building larger models.

The Hype is Real

ChatGPT took the world by storm when it was released in November 2022 because it was the first generative AI model that was over 100bn parameters and showed a wide variety of emergent capabilities. It also had a simple and intuitive interface and had been fine-tuned specifically for chat.

But ChatGPT is far from the only generativeAI model that has seen a huge amount of success. Below is a list of some of the other generative AI platforms that have gained a lot of traction:

Claude 2 - a chat model, similar to ChatGPT by Anthropic.
LLaMa 2 - a large language model that has been publicly shared by Meta.
Stable Diffusion - a test-to-image model.
Bard - a chat model, similar to ChatGPT by Google.
Dall-E 2 - a test-to-image model.
Midjourney - a test-to-image model.
Jasper.ai - a model fine-tuned for marketing applications
Character.ai - a platform for creating different characters you can interact with.

So why is there so much hype around generative AI? A big reason is that generative AI models have now got to the size and capability where they are pretty good at a wide variety of things. This is a huge difference to previous artificial intelligence models that were very good at only one thing. But what is the impact of this going to be?

https://www.youtube.com/watch?v=PGpMao4VLoc

Carl Frey, the Director of the Future of Work at the University of Oxford summed this up perfectly in a webinar when he said that generative AI would reduce the barriers to entry for knowledge work in the same way GPS and Uber had reduced the barriers to entry for taxi drivers. This idea is really exciting because it will be the first time that AI has been able to disrupt knowledge work, which he hopes will lead to people being able to accomplish much more with their time.

Another reason there has been a lot of hype around generative AI is because of the huge amount of investment and resources being thrown at the technology right now. And much of this is in the open source community, thanks to the leak of the LlaMa model from Meta.

Prior to March 2023, building Large Language Models (LLMs) was the domain of large, resource rich companies. For example, GPT-3 cost c.$4.6m and 30 years of processing time to compute. That all changed when Meta’s GPT-like LLM model was leaked to the open source community a week after it was officially release. This led to an immediate levelling of the playing field with many open source developers now able to run LLaMA locally. This greatly accelerated and diversified generative AI development and has led to a huge proliferation of new generative AI platforms, products and services.

The generative AI open source community has been moving at such a pace that it raised a huge amount of concern with engineers at Google. There was a famous thread on Twitter discussing this and predicting that the open source community would win out over time.

Where the open source community is absolutely excelling is getting more and more capabilities out of smaller models. Where this becomes fascinating is I think within the next year we’ll see models that are small enough to fit on a mobile phone and have a similar capability to GPT-3.5. This could be a game changing, especially in countries where mobile internet connectivity isn't great. We’ll soon have a huge knowledge base on our phones without having to rely on an internet connection.

Summary

This brings us to the end of the second instalment in our Classroom series where we’ve introduced Artificial Intelligence. We’ve covered why generative AI has suddenly become an incredibly important technology in 2023, we’ve taken a deep look at OpenAI and ChatGPT, covered other generative AI platforms and demonstrated the importance of the open source community.

In the next article we’ll be covering some of the recent research around generative AI and looking at the impact that the technology is already having on consumers, businesses and society.

An Introduction to Artificial Intelligence: Part 1

Sean 🤓 — Sun, 30 Jul 2023 15:43:30 GMT

What is Artificial Intelligence?

Hi, and welcome to the first post in our Classroom series that covers an Introduction to Artificial Intelligence.

Artificial Intelligence is a term that has been around since the mid 50s. It was defined as a field of computer science that's focused on creating technology that can perform tasks associated with human intelligence, such as learning, reasoning, and decision making.

“A field of computer science focused on creating technology that can perform tasks associated with human intelligence, such as learning, reasoning and decision making”

This term was coined by a team of researchers in 1956 that built on previous work that was done by Alan Turing.

75 years of AI Development

AI development started back in 1947 when Alan Turing gave the first public lecture to mention computer intelligence, but the term artificial intelligence wasn't coined until 1956. As you can see in the timeline above, there was quite a lot of work done through to the mid 60s.

Whilst there was ongoing work in AI throughout the 70s, 80s, 90s and early 2000s it wasn't until 2011 when IBM Watson was released and appeared on the American TV show Jeopardy against some of the champions that AI re-entered the public imagination. Watson won first place on Jeopardy winning a prize of one million dollars and grabbed lots of headlines in doing so.

Since then there has been a rapid amount of development in AI and over the last 12 years there has been a lot of progress. Some interesting milestones have been:

📍 In 2017 Google released a technology called the transformer architecture, and that was absolutely revolutionary. It was a pivotal piece of technology and underpins all the advancements in Generative AI that we see today.

📍 In June 2018 OpenAI released GPT-1, the first of a kind foundational model. It had 117 million parameters, which compared to today's sizes is incredibly small, but already showed a lot of promise.

📍 In October 2018 Google released a similar foundational model called BERT, which was a similar size to GPT-1.

📍 In February 2019 GPT-2 was released which had an order of magnitude larger number of parameters at 1. 5 billion.

📍 In May 2020 there was another big leap in the size with the release of GPT-3 which had 175 billion parameters.

📍 In November 2022 ChatGPT was released by OpenAI based on GPT-3.5 which had been fine-tuned for chat with ‘reinforcement learning with human feedback (RLHF).

Main Branches of AI

There are six main branches of Artificial Intelligence:

These different branches are really split into two main areas, the ones that deal with structured data and the ones that deal with unstructured data.

Structured Data

Structured data is data that is in a format that is very easy to read by computers such as tables.

Unstructured Data

Unstructured data is where a lot of progress has been made over the last 10 years. Unstructured data is things like long passages of text, videos or images. It’s data that doesn’t have an inherent structure to it that computers find hard to understand.

The domains of AI that deal with unstructured data have now got to the point where it can deal with it in a very, very sophisticated way. This progress underpins the advances we’re seen in Generative AI.

What Is Generative AI?

“Generative ai refers to a category of artificial intelligence (ai) algorithms that generate new outputs based on the data they have been trained on. Unlike traditional ai systems that are designed to recognise patterns and make predictions, generative ai creates new content in the form of images, text, audio, and more.”

I think it says a lot that the official definition above from the World Economic Forum was only defined in February 2023. Generative AI is, for the first time a computer system has been built that is able to generate content on its own, based on what it's been trained on.

Having said that though, it's important that we put the term artificial intelligence into context for what it actually is, because if you go back to that definition from the 1950s, it's a very, very broad definition.

Applied Statistics

ft.com

So what really is Artificial Intelligence in its current form? Well Ted Chiang, the acclaimed author had a fantastic response when asked about this by the FT. When asked what he would call Artificial Intelligence if he had to invent a different term for it his answer was instant: Applied Statistics.

This term is really important to keep front of mind when thinking about Generative AI. None of the platforms or models are conscious, they don’t have intelligence per se, they’re just really really good at maths.

Generative AI leverages two main branches of artificial intelligence that we covered earlier:

Neural Networks

Neural networks are a technology that is designed to work like the human brain, and is made up of millions of artificial neuron’s that are all connected to each other.

Natural Language Processing

Natural language processing is a technology that helps computers understand human language. It enables people to interact with computers and digital devices in a more natural and intuitive way. You will likely have come across a lot of this technology over the last five years or so with devices like Alexa, Siri and other voice assistants.

Terminology

New technologies bring new terms that need to be understood so we can truly understand how the technology works. The four terms that are important to understand with Generative AI are Large Language Model, Tokens, Parameters and Dimensions:

Large Language Model

A Large Language Model (LLM) is a type of machine learning model that is trained on a vast amount of text data. It uses this training to generate human-like text based on the input it receives. Examples of LLMs include GPT-4 from OpenAI, Bard AI from Google and LLaMa from Meta. They can perform a variety of natural language processing tasks such as translation, summarisation, answering questions, and more.

Tokens

Tokens can be thought of as pieces of words. And for some bizarre reason, a token is about four characters in English which is on average about three quarters of a word. GPT 4, the latest model from OpenAI, can ingest a prompt up to 32,000 tokens long, which is approximately the length of the book Animal Farm. A huge amount of content can be ingested in a prompt when you are asking questions of a Generative AI model.

Parameters

Parameters are talked about a lot in the Generative AI space, and they're more like a model's knowledge or memory from all the data that its been trained on. GPT 3 has 175 billion parameters, which is like someone having read a whole library of books and being able to remember absolutely everything. These models have a huge amount of ‘memory’!

Dimensions

Dimensions are like different aspects, features, or meanings of a word. So in the above example, if you took the word ‘intelligence’, it has similarities to concepts such as ‘problem-solving’, ‘understanding’, ‘knowledge’ and ‘learning’. Dimensions allows a large language model to understand how close a word is to other words or concepts.

Dimensions are represented by what's called a vector of numbers. GPT-3 uses 768 dimensions, which are adjusted by the model during training to help it accurately predict the next word in a sentence.

How Generative AI Works

At a very basic level the above diagram is how generative AI works. In this example it takes a text input and it turns that text into a numerical representation, in the example of GPT-3 a vector of 768 dimensions. The generative AI platform then feeds that numerical vector into the large language model.

The large language model will then output an image or some text or video or any other type of content that is being built to output. The way that it does this is iterative. The way a large language model outputs text and the way that it outputs images is slightly different, but the broad principles are the same.

In a text-based large language models it just predict what it thinks the next best word should be in a sentence, based on the context of the prompt that it’s been given. So, in the example above, if you gave the model the word ‘there’, it will then predict that the next likely word is ‘are’, the next likely phrase after that is ‘lots of’, and then after that it predicts ‘reasons’.

So, the model produces the sentence ‘there are lots of reasons’. This is a very, very basic example and it’s actually something we’ve seen a lot of over the past 5 years or so:

Anyone who has a modern mobile phone, whether it's On iOS or Android will have seen text prediction on their device. Text prediction is powered by a large language model that is trained to predict what the next word is that you're typing in a sentence. So these large language models have been in our pockets for years now, they just haven't had the sophistication of the large language models that we now see powering generative AI.

Summary

This brings us to the end of the first in our Classroom series where we’ve introduced Artificial Intelligence. We’ve covered a comprehensive overview of the concept and history of AI, outlined the six main branches and delved into what Generative AI is and how it works.

In the next article we’ll be covering why Generative AI has become such a hot topic recently, go into more details on what ChatGPT is and look at why this area of technology is so hyped at the moment.