For a change, rather than looking at the past, today’s post is rather topical, and looks at a subject that at first glance may seem irrelevant to the blog – Artificial Intelligence, or AI.
When I write a post for the blog, I try to apply a degree of intelligence. I use a wide range of sources – books (old and new), maps, national and local archives, old newspaper archives, libraries, academic resources, visits to the site that is the subject of a blog post, and hopefully, some knowledge of London built up over the years.
I aways try to find more than one reference to key information about a topic, and only use a limited number of reputable sources on the Internet, for example British History Online.
Fortunately, I make very few mistakes, and when I have, I am very grateful to readers who have pointed this out via a comment or email, so I can correct.
The numbers of visitors to the blog has grown considerably since I started in 2014. Most of these readers come via Google, but over the last few months, I have seen that the blog is being accessed by AI services such as ChatGPT and Google Gemini.
There is much noise about Artificial Intelligence. Depending on what you read, AI is either going to save the world, or take everyone’s jobs and destroy the world.
Most of the end user AI services such as ChatGPT, Google Gemini, Microsoft Copilot and Grok (X was Twitter) are conversational based systems using extremely large collections of data.
Conversational, in that they attempt to provide the user with the impression that they are talking to another human, and that you can have a conversation with them. The large collections of data that they use to build a response, are from anything they can search – websites, blogs, archives, photos, books etc.
As someone who has written getting on for 2.5 million words in the blog during the last eleven years, I wanted to know what impact AI would have on this form of writing? Was AI using information that I had created, how accurate AI could be, and whether AI could replace a blog as an information source, and if so, how accurate would be the response.
So I thought I would give it a try.
I used a few of the most high profile AI tools to ask a simple question about one of my favourite places.
Imagine you are walking from St. Katherine Docks to Wapping, and as you walk along St. Katherine’s Way, you find an intriguing set of river stairs – Alderman Stairs – and you want to know their history, so you ask a few AI tools “What is the history of Alderman Stairs”, (you can find my post on Alderman Stairs, here) and this is what I did, starting with:
ChatGPT
ChatGPT was one of the first readily available AI tools, with the first release in November 2022, and the tool is currently on the fourth version. Chat, the first part of the name should be self explanatory, GPT stands for Generative Pre-Trained Transformer, which basically means it captures information from the Internet in order to learn how to respond to a question.
You can try out ChatGPT by clicking here, where you are presented with a simple “What can I help with?”, to which I asked the following question:
ChatGPT provides the answer to your question in a response screen, but also allows you to copy the answer as text, so in theory, I could use the answer to write a blog post, so the following text is an exact copy of ChatGPT’s answer to the question “what is the history of Alderman Stairs?“:
“Alderman Stairs is a historic set of stairs located in the City of London, connecting Upper Thames Street to Puddle Dock and the River Thames. The stairs have a rich history, and their name, along with their structure, reflects the changing history of London.
Origins of the Name:
- Alderman Stairs is named after an alderman, a senior elected official in the City of London. In this case, the name likely honors one of the city’s aldermen who played a significant role in its development or public works in the area.
- The exact alderman behind the stairs’ naming is not fully documented, but it was common in the medieval period for prominent citizens to have public infrastructure named after them, especially in riverfront areas.
Development and Historical Use:
- The stairs have long been part of London’s waterfront, providing access from the land to the river. They were essential for the transport of goods and passengers during the times when river trade was crucial to London’s economy.
- The Thames served as a bustling trade route, and stairs like Alderman Stairs were crucial for loading and unloading cargo. Ships would dock at these stairs to transfer goods such as coal, timber, and other materials necessary for London’s growth.
The Changes Over Time:
- Medieval and Early Modern Period: Alderman Stairs likely existed in some form by the late medieval period when river transport was key to London’s development. Stairs and wharves were often maintained by wealthy and influential individuals or companies.
- Victorian Era: The industrialization of London in the 19th century, along with major changes in the River Thames, led to the construction of larger docks and piers. Over time, this reduced the prominence of stairs like Alderman Stairs for commercial activity.
- 20th Century: By the 20th century, many stairs and docks along the Thames had fallen into disuse due to the decline of river trade and the increasing use of road transport. Alderman Stairs, like many others, would have lost much of its original commercial purpose.
Modern Day:
- Today, Alderman Stairs is primarily of historical interest and is considered a heritage site, reflecting London’s long relationship with the River Thames. The stairs themselves are located near the London Bridge area, in a part of the city that has been significantly redeveloped.
- Though the area has been extensively modernized, the stairs remain as a reminder of London’s medieval and early modern history. They have been restored to some degree and are occasionally used for public access to the river.
The history of Alderman Stairs highlights London’s transformation from a city dominated by river trade to one marked by modern infrastructure and development. Despite these changes, Alderman Stairs remains an interesting piece of London’s architectural and historical legacy.“
I was really stunned with the amount of issues within this response.
Firstly, it reads like the answer someone would give on a school or university exam paper, where not enough revision has been done, and you have skim read at the last moment, and attempt to put together a reasonable answer using high level and half remembered facts (or was that just me!).
Secondly, the first paragraph on the location “Alderman Stairs is a historic set of stairs located in the City of London, connecting Upper Thames Street to Puddle Dock and the River Thames.” Alderman Stairs are nowhere near the location provided by ChatGPT, although I was concerned I may have missed something when writing about Puddle Dock, so went back and checked lots of maps, books etc. The following is Rocque’s 1746 and shows the street Puddle Dock (red arrow), Puddle Dock (yellow arrow) and where Puddle Dock accesses the Thames (blue arrow) – I could not find any reference to an Alderman Stairs anywhere near Puddle Dock:
Thirdly, the section on the origin of the name Alderman Stairs includes the following “The exact alderman behind the stairs’ naming is not fully documented, but it was common in the medieval period for prominent citizens to have public infrastructure named after them, especially in riverfront areas.“
There are some clear candidates for the source of the name, dating not from the medieval period, but from the late 17th / early 18th centuries.
As ChatGPT is a conversational tool, it offers the option for a follow-up question, so I asked “are you sure this is correct?”:
ChatGPT is certainly polite, as it apologised for the confusion, and confirmed that I was right to question the answer it had previously given “as there seems to be some inaccuracies in the historical details”.
In ChatGPT’s revised answer, the location of the stairs has now moved, and they are now “specifically close to the Cannon Street railway bridge”. I checked Rocque’s 1746 map, when obviously Cannon Street railway bridge was not there, the bridge started across the river where Steel Yard Wharf is shown in the centre of the following map:
There were no Alderman Stairs in the vicinity, or in later maps, including mid 20th century OS maps.
The rest of the answer is just as high-level and vague as the first answer. So my next question was “is there an information source for alderman stairs?”:
At first, my ego was somewhat deflated as my blog page on the stairs was not referenced, but on second thoughts I was rather pleased not to be associated with the quality of the ChatGPT response.
Even the list of sources raises some serious questions.
There are two books listed by Stephen Inwood, who has written a number of excellent books about London. I have the History of London, but I cannot find any reference to Inwood having ever written a book called “A history of the City’s Roads and Their Names”.
Likewise, the book “Old London’s Rivers and River Stairs” by E.S. McLachlan sounds a fascinating book, but again I can find no reference to such a book or author. I looked for these books at Amazon, Abebooks, the British Library and London Library, as well as a general Google search, but no luck.
I would be interested to know if these two books actually exist as I would like to have a copy.
In the Online Sources section, the Londonist website is listed. I tried a search at Londonist and there were no references to Alderman Stairs.
One of the tools provided by ChatGPT is image creation, and if you use Social Media you will find it flooded with images created by this type of AI tool.
I could not resist seeing what ChatGPT thought Alderman Stairs looked like, so I asked it to “Create an image of alderman stairs”, and this was the response:
A remarkable interpretation. I cannot argue with the “historical charm and tranquil atmosphere” of the real Alderman Stairs, and this is what they really look like:
By now, I was thinking that I was being a fit unfair with ChatGPT. Alderman Stairs is a rather obscure topic, and it would be reasonable for the tool not to be aware of such a place, although if that was the case, why does it just say that it does not know, rather than cobbling together a false answer. Part of intelligence has to be admitting when you do not know, rather than pretending that you do.
To see if I was being unfair to ChatGPT, I put the same question to other AI tools, the next was:
Microsoft Copilot
Microsoft Copilot has now got the correct location. It also provides a precis of the story of the stairs. The information is referenced, and at the end it provides the references, and links if the reader wants to know more.
The most used reference is to my blog, and a comparison of my blog post on the stairs with Copilot’s response shows where the information has come from, although it is a very high level summary.
I can see where this approach would be useful, as a quick way of finding information sources for a topic. Ask a question, and Copilot will provide a summary with a list of sources for follow-up.
I then put the same question to:
Google Gemini
The answer provided by Google Gemini is basically a summary of some of the key points from my blog post. At the end of each sentence, there is a button, and clicking provides the source of the information, as shown in the above example, and also in the example below:
Two points regarding Google Gemini’s response:
Firstly, the way these tools summarise gets rid of much of the context. In the above example there is a sentence on Irish immigrants seeking a new life in London.
In reality, these were very poor Irish people, probably close to starvation, and in the following screenshot from my blog post on Alderman Stairs, I have included the extract from the Illustrated London News which reported the landing at the stairs, and provides much more context:
My second concern is that when you click the dropdown box for the reference, Google Gemini states “Google Search found similar content, like this”. It is not “similar content” it is the original source information which Google has copied to use within its AI tool.
Getting rather depressed by now, I thought I would try one final AI tool. This tool is part of X (the old Twitter):
Grok
Again, some good information, but summarised and without the full context, for example, the Illustrated London News report I quoted in my blog post about the Irish Paupers is in the Grok response as “These stairs have been busy, with accounts mentioning up to 1200 people in one shipload, suggesting a high volume of traffic similar to that seen at Tower Bridge today” – and I have no idea why Tower Bridge is included as a comparison with Alderman Stairs.
Grok has a list of “Relevant Web Pages”:
Grok calls this listing “Relevant Web Pages”, where in reality, these are pages and websites where Grok has sourced the information to compile the response, including my blog.
This very quick look, using a single question, raises a number of questions:
Copyright
AI systems are trained on data which is electronically available and much is sourced by searching the Internet. AI tools then use the information found to build a response to a question and provide this as the AI’s answer to the question.
AI systems are therefore using the work of other people, authors, and organisations, and where the source is given, with Google it is quoted as “similar content”.
Microsoft Copilot was the best of those tested in providing links to the sources used to build a summary response.
At the moment, if you search for Alderman Stairs on search engines such as Google, you will get a link to my post. As AI tools improve, they will end up showing a comprehensive answer, thereby reducing the incentive for people to find the original or alternative sources, so AI tools will use data from other websites whilst at the same time reducing the visibility of the sites which provided the source information.
Currently, the UK Government are consulting “on proposals to give creative industries and AI developers clarity over copyright laws”.
The key points from this consultation are:
It will be interesting to see how this develops, however there is not a good track record in the protection of data on the Internet, particularly where the big tech companies are concerned.
Historical Accuracy
The use of these tools means that there is far more risk that information becomes distorted, provided out of context, or is just plain wrong.
It is also easy to see how the response from AI tools can be manipulated as they are basically building a response from the information they have found at other sources. If those sources are using false information, AI tools may probably just repeat this.
Much of the ChatGPT response was just so wrong, and users would have far more confidence in the output of these tools that when an AI tool does not have the information, it just answers with an “I do not know”.
I have a very amateur interest in London’s history, do this for my own interest, and fortunately others find my content interesting as well, however for anyone who writes professionally, depends on writing for their income, carries out academic research etc. I would be concerned about where this is going.
ChatGPT did however redeem itself with a final test. As I was finishing this post, I thought it would be interesting to see what ChatGPT knew of my blog, and this was the answer:
I really like ChatGPT’s summary, and it is a far better summary of my blog than I think I could write – so you can see where Artificial Intelligence can be useful, and I might use the above text as my new “About” page.
Artificial Intelligence is not going away, and if you would like to try out these AI tools:
- Click here for ChatGPT
- Click here for Microsoft Copilot (scroll down to the bottom of the Ads)
- Click here for Google Gemini
- Grok can be found if you have an X account
Whatever the future of AI, I can promise you that all my blog posts will always be written by a human, using old books, maps, library and archive research, newspaper archives, photos and images, visits to site etc. and will come with the poor grammar, punctuation and occasional typos, that comes with keeping up the amateur production of a weekly post, and as ever, I really appreciate any corrections.
Quite.
An excellent analysis as you are the training source for the responses and can easily spot the errors. But anyone innocently seeking knowledge is going to be seriously waylaid by the dodgy information served up. There is something to be said for the ‘traditional’ Google click on the top resources.
Now extend what you have found to every other query and the accumulation of false or misleading information multiplies tenfold.
AI isn’t intelligent, but a brilliant machine learning series of algorithm that is great for pattern recognition but very weak on understanding – because it can’t.
I think your blog is safe!
What a fantastic summary of the new tools available and their limitations.
Thank you! I learned more from this than the plethora of articles discussing AI in general.
Same here!
I tried the same experiment with my London history-based PhD topic–which I chose because I know enough about the topic to spot inaccuracies. The result was laughable. The information provided was not only shallow, but much of it–even basic stuff such as dates–was wrong. I messed around a little to see if I could improve the results but basically it was always dross. It bore as much relation to history as 1066 and All That, only without the humour.
The idea that AI can be a substitute for well-conducted human research is simply not to understand what research, or for that matter intelligence is.
Right now I am involved in a project about Taiwan and I willingly admit that AI machine translation is an absolute boon, given how slow my Chinese reading is. But that’s the point, a machine can translate, because languages follow rules and so do machines. When it comes to judgment, insight, consideration, a sensitivity to irony, machines can’t do that. In fact, AI isn’t even very good at vetting sources. It often seems to think stuff trawled from a Year 9 history essay is as worthy of citation as a text by a Regius Professor.
The idea that AI can produce essays of the quality of those in A London Inheritance is frankly laughable, and anyone who would argue to the contrary is someone who obviously doesn’t know what research really is.
I thoroughly agree
An interesting experiment. Absolutely, AI lacks “common sense”, let alone the ability to operate in a nuanced fashion. What is really concerning is that all these crappy searches are adding to the existing content which in turn will be trawled by AI, leading to the quality of information becoming more and more diluted.
I really appreciate your blog and and the care and attention to detail of your research.
As fascinating as one of your posts or tours.
I’ve just started using AI (as I enter my 7th decade) for the structure and general standardisation of documents at work. The content of the AI tools I’ve used so far falls well below what my brain can more accurately fill in..
The tool’s ability to edit or analyse content that the user gives it, is probably a better and safer application rather than using it as an advanced form of search engine. As your summary example shows the tools can write well and many people use them to manipulate text that the user has supplied. You are right to point out that one of the challenges is that the tools are in many instances stealing content ie material that should be subject to copyright.
Hallo. Thank you for your very insightful analysis. Yes, it looks like AI is only ‘half a steam-engine’ at the moment, but I’m hoping the future will be more golden. My current (!) main concern is that AI is not looking in all the corners and we don’t know, for sure, where and what the barriers are. I am sure there will be, and most likely there are already, plenty significant barriers to data either because it is behind a paywall or the data protected for very good and security reasons. Perhaps the data analysts haven’t found a way in yet . I think knowing what is inaccessible to AI is as important to understanding what is, so thank you for your article. The more we know the safer and better focussed we will be.
This article is absolutely brilliant. I did a test on Chat GPT a while ago, but didn’t follow through like you have. I will give it another go and also check CoPilot etc..
A very interesting and (as usual) well presented article. It was interesting that Microsoft Copilot did a good job. You should be aware that the latest Microsoft Windows 11 AI component includes a system called Microsoft Recall. This ‘service’ takes and stores a continuous series of screenshots to ‘recall’ your previous work. These screenshots include *all your personal data* including dob, NI numbers, usernames etc. You might see this as a major data security risk (I would!) and I suggest a quick search of YouTube on how to ensure this intrusive mechanism is switched off. It is usually on by default where Microsoft AI features are used. I hope this helps.
….and your blog will also continue to be highly regarded (and valued) by all those “humans” (aged or otherwise) that really enjoy the knowledge and insights that you provide.
Long may it continue !
Terrific; post, thanks. I also searched for “Old London’s Rivers and River Stairs” by E.S. MacLachlan on my favourite book database and nothing came up. Whether the book exists or not, ChatGPT must be referring to a digital copy to which no-one else has access.
If you point out that it must be lying, it gets polite to the point of grovelling!
At archive.org there’s an obscure book you might know of that I lost in a house move:
https://archive.org/details/historyofsignboa0000jaco/page/n3/mode/2up
This is an outstanding, in fact important, summary of what large language models can and can’t do at present. One of the many weaknesses of “tech” is that there is and always has been a lack of criticism, or willingness to be criticised, and your work is unusual.
I shall be referring to it at work, as there is a big AI push there and I have been doing a similar analysis with roughly the same conclusion as yours (summarising an [informed] single source = excellent; deriving a summary from multiple sources = downright dangerous, to the extent of “confabulations” to fill in gaps i.e. lying).
This is an extremely useful summary of different AI tools. Thank you very much for taking the trouble to provide it for those of us who don’t know about these new developments, and for warning about the risks.
The best comparative analysis of the AI tools that I’ve seen and confirms my concerns. This was very informative. Thank you !
All historians, professional or amateur, now need to be aware of AI and its impact on research – this is a great example of being alive to the latest technology and being able to quote your own sources with confidence!
Good work and a very timely and useful lesson for us all.
Thank you for this very interesting report. I haven’t researched or experimented with AI myself, so the results of your own research have been most illuminating. Clearly, AI is in its infancy; its quality, from the standpoints of accuracy and reliability, leave a lot to be desired. I’m less concerned about what it omits than I am about its ability to present inaccuracies as fact, concerns having also been raised recently about significant AI inaccuracies allegedly being published as fact in certain news media. We are all used to the speed with which changes and developments occur in IT, and we can hope that in the not-too-distant future the accuracy and reliability of AI will be improved. In the meantime, it’s a good thing that its current shortcomings be highlighted so that we may all be on our guard against electronic falsehoods; since perfection is likely to be at least as difficult to achieve among AI-generated reports as it is among those of human origin, we may always need to be told when data are derived from, or through, AI sources. Alternatively, we may need known reliable sources to carry some kind of quality mark to assure us that the originator of a report, and by implication the veracity of the facts contained therein, can be relied upon at least as much as we’ve relied in the past on reports published in books and papers from trusted sources. As we are currently being bombarded with widespread deliberate misinformation and “alternative facts”, however, some of which emanates worryingly from sources we have traditionally tended to trust, including from those who control published Standards within society, I feel that the verification of facts may, from now on, be something of a challenge.
Absolutely brilliant, thank you.
But then I wouldn’t have expected anything else
Excellent analysis!
I so appreciate the work you put in and enjoy immensely what comes out of
A superb dive into AI. You did a thourough test and identified the weaknesses of these tools. I do fear the ‘high level’ narrow attention span and ease of these tools will further contribute to the decline of we obsessively curious rabbit hole seekers, and yet, may allow more people to be curious to expand on the simple answers these fake brains offer. Who knows, but it will be interesting, for sure.
Thank you for all your hard work, old maps and all that real human (RH) stuff.
RH vs AI . I love it !
What a wonderfully researched lesson in the pitfalls of AI . For my own part , I write simple field reports for birdwatching and field naturalist clubs I belong to . It’s interesting to ask for an AI version after writing my own . I find discrepancies all the time especially in species names . Which is the entire point of my report . I do get the literary equivalent of ‘soft and fuzzy ‘ illustrations of the areas we visit , almost tourist brochure descriptions but that’s not what I require . I fear that we are at the tipping point in the dumbing down of society as we can now only process a small part of the vast knowledge instantly available to us at the click of a mouse or the tap of a screen . Thanks for the work you put into your posts . Sitting in my rather obscure part of rural Australia , it keeps me connected to an earlier heritage of my family in Old London .
Very interesting, and thank you for doing all the research to check it! I have a deep mistrust of A1 so this is a concern of mine.
I wouldn’t say all things pre-AI are historically or otherwise accurate. There are lots of mistakes and misinterpretations. The London Chronicle surely got some things wrong. I am worried human minds will become mush with this higher level of bringing data together and hopefully people will challenge and reinterpret information, but doubtful. I would imagine lots of government and business reports can be easily generated but I have doubts there will be people knowledgeable enough to spot errors or scrutinise results. There is nothing like having to do the research yourself to understand it.
I’m afraid the further we get from nature and earthly materials the less we are truly alive and I suspect the primary objective of AI is profits, power, and control. Storytelling and art from creative individuals will cease to be important. I prefer traditional knowledge and understanding of engineering and architecture but maybe AI is a good benchmark for comparisons. AI has potential within medicine and science to bring huge amounts of research and information together but there still needs to be experienced, discerning, and intelligent people to understand and differentiate between what AI produces and what is experience and truth. I think many institutions and people will accept whatever computers spewsout and human intelligence and life experience will decline. People will not appreciate individual learning and knowledge. Easy spoonfed information will have little meaning or value without people participating in its generation. Strange world and the big companies behind AI and academia and governments are not able to influence or control it.
My last point is the continual upwards trend of generating computer data surely has environmental implications as more and more computers and storage of data use more and more electricity. Will it ever end and how long will electronic records be able to be kept before they’re corrupted or just lost in time. Shame, but I think AI though technologically incredible, will do little to enhance or enrich our life experience.
There are many variants, but the accepted version nowadays is ‘Saint Katharine Docks’, with the local area known as ‘Saint Katharine’s’. Always with an ‘a’ and not an ‘e’.
Really really intersting and so pleased to see you put the various platforms through their paces. I cannot conceive of abandoning real people and their writing styles for anything artificially generated.
A very interesting post. I’m amazed how different the results from different AI bots were. Love your blog, especially as it relates to my family history, and it has helped me to build a picture of how my ancestors lived over the past 400+ years. I’m so grateful you have published all this.
The AI experiment was illuminating! Very interesting to see the differences between these AI apps.
I haven’t played with the ones you tried, I don’t have the time at the moment and I’m worried that it will know who I am and keep suggesting nonsense! I used to teach computing so I don’t want to go down a rabbit hole with AI while I have other things to do.
Avery interesting read about AI. I’ve used Copilot and agree with your points about it.
I do however have concerns about copyright issues and AI.
Fascinating piece, and serious research in its own right. As ever, thank you for your continued commitment to giving us such a thought provoking and well informed series.
I really enjoyed reading your critique of AI tools used in this context . As always ,you take a very methodical and clear approach to the subject and in doing so shed valuable light on the topic . All power to your pen !
Really interesting. I asked Chat GPT what it thought of this post and it’s response can be found here: https://chatgpt.com/share/678626e0-44d8-8000-9ecb-3325686c998b
Incredible! This is too fun. It may be like the shadow asking the echo to review the fragrance, or something. I am so confused I feel like Alice stuck with the rabbits. We mere humans need only stand by and watch as it all passes by. The cloud and all this mayhem may just vanish one day and we can return to our simple oral folk tales passed on by the fire. Kinda bizarre, for sure.
I (69) ,and rather rooted in analogue ways of media uses, have always wondered, how these AI tools do function, and couldn’t for a long time figure out the way
results came about , but with this fabulous blogpost where you can compare all relevant texts and sources I get the gist of it all and that is why I’d like to call it for myself a true moment
of enlightenment.
Can’t thank you enough for that.
Günther
What a huge amount of research you do! thank you. Something that astonished me
particularly was the historic amount of Irish migrants that arrived in the heart of London,
comparing that information with ever-more alarming newspaper reports of illegal migrants
who arrive by sea on UK’s South Coast. As always with reading history, one sees repeating
patterns of human behaviour. Long ago I was reading
Apologies: my comment went off in the middle of composition – not to prolong discussion, thank you very much for your immaculate research and I so agree with your conclusions and many of the follow-up comments by readers.