The Transformer: Origins and Future in Generative AI

Jakob Uszkoreit, a key figure in the development of Transformers, discusses its origin, impact, and responsible deployment in generative AI.

### The Origin of Transformers and the Future of Generative AI: A Conversation with Jakob Uszkoreit
* 🧠  Jakob Uszkoreit, a key figure behind the original idea for Transformers, explains how the concept was developed to **overcome limitations of existing neural networks**.
* 💡  The 2017 paper on Transformers was seen as **one step in a series of advancements**, with early signs of its potential across various domains already apparent during development.
* 🤖  Uszkoreit discusses the challenges and benefits of deploying AI models in real-world products, highlighting the **importance of responsible implementation**.
* 💊  The conversation shifts to Uszkoreit's current venture, Inceptive, a biotech startup leveraging **generative AI** to design RNA molecules for medicine.
* 💰  Uszkoreit sees profitable applications of generative AI in industries like energy exploration and biopharmaceuticals, emphasizing the **potential for significant value creation**.
* 🔮  He addresses concerns about bias in AI systems, stressing the importance of **data curation** and **responsible deployment**.
* 🎓  When asked about education for future careers in AI, Uszkoreit recommends a mix of **natural sciences, mathematics, and computer science**, rather than a specific major.
* 🌐  Uszkoreit believes that despite the energy consumption of AI training, future advances in energy technology will make the **overall impact positive**.

Our guest today, Jakob Uszkoreit, is often credited
with the original idea for Transformers: the foundational concept that makes generative AI
possible. At Google in 2017, he co-wrote a paper about the idea
with seven other colleagues. Then in 2021, after more than a decade at Google, he
left to start Inceptive, a biotech and AI startup with backing from Nvidia. Here's our conversation. Jakob, thanks for joining us today. Thank you for having me. Absolutely. I have heard that the original idea for
Transformers started with you. Is that true? And where did it come from? Well, let's see. In science, I would say there are rarely, if ever, truly original ideas. It's you're always standing on shoulders of giants. But really what happened around that time is that my
group was faced with an interesting conundrum where we had fairly advanced
and expressive

deep neural networks called LSTMs, long short-term
memory networks, in a configuration that was dubbed Seek-to-seek, also
developed at Google around 2014-2015, that ultimately were really showing state-of-the-art results in neural machine translation or in machine translation overall. Yet complexity, their training speed was so slow that
if you had enormous amounts of training data for a given
problem, you would never actually be able to train even on a fraction of the training data that you had
available. And as a result, if you use those types of neural
networks that were in principle more powerful and more expressive than much simpler, say, feedforward
architectures, you couldn't train on as much data, and as a result, would remain inferior to
much simpler networks trained on a larger fraction of the available data. And so, inspired by that problem, really, we looked at
existing mechanisms or architectures that could help us
overcome this limitation where you would both end up

with something similarly expressive to these RNNs or
LSTMs, while actually massively accelerating training times and training
speed on the kind of accelerator compute that we had at the time. That is still quite strikingly
similar to the kinds of accelerators that we have today. And so that led to, and this is maybe an idea
that I contributed here, this idea that we could use a mechanism called attention
mechanism that had been published and proposed in the two years before we
actually started working on the transformer, maybe even three years before that. Use
such an attention mechanism, not to have one signal informed by information in another,
but actually use it in a way that we then called self-attention, where you basically
have a given structure, a signal, say, a sentence or so, attend to itself in order to further
the interpretation or understanding or ultimately vector
space representation that you have of it. And that mechanism is one that fits exceptionally well
onto GPUs.

And so that's then ultimately, maybe if you wish, one original idea among many, many others that gave rise to what eventually ended up being called the transformer architecture, that we then published in 2017. When you were working on the paper, did you have any idea that it was as monumental as we now see it to be? It really was one step in a long sequence of steps on this road. My group had published a paper just a year prior to that, on a model called the Decomposable Attention Model that actually, by and large, was structurally quite similar, but not nearly implemented as well, and not nearly as fast as the transformer. So really, I think we're making a mistake if we look at it as monumental. But then at the same time, we thought if we could really strike this balance effectively between exploiting the compute resources that we had or maximizing the expressive power in a neural network.

architecture that you can support with that flavor of
compute, then that could affect not only, you know, the classic or typical
applications of these neural networks at the time, but open up many, many new ones. But there is a sentence that we've obsessed over for
quite a while about how excited we are to apply this to a variety of other problems, and also other types of
problems going forward, and that we have pretty high hopes for that. Truth is, at the time, we had already started doing
that and had preliminary results and or at least hints of results that would suggest that, yes, indeed, you
can apply this to a broad variety of problems, ranging from things like protein structure prediction all the
way to computer vision. The consumer-facing ChatGPT Revolution is one side
of the coin. But were you already implementing these things at
Google, Google search, Google Assistant, Google translate, those were teams
you all worked on, right?

Yep, that's right. We had some prototypes. Uh, that would apply Transformers in, you know, even
in the very early days. In some of these products, it took a very long time for those to really
reach the light of day. ChatGPT hasn't actually been around as a consumer
product for that long. And so there are actually applications, for example,
at Google and other places where transformers have been deployed in production long before, but also to much,
much less fanfare. Really, it was ChatGPT that shone the spotlight on
these applications, because there's not that much more to the product than really a slick interface
around a very, very well-trained transformer trained on large amounts of data with
lots of compute and of course, engineered super. Well. What is your reaction to the perception that Google had
this wonderful idea internally and let it slip away and has fallen behind? I mean, ultimately Google has grown into, you know, a
large company, and they're not

necessarily notorious for being particularly nimble at
the same time. It's not actually the case that it really took Google that much longer to launch
this in certain products. I feel what we're actually seeing is that
maybe Google here hasn't been able to be as daring as, you know, a much, much
smaller companies such as OpenAI when it comes to applying this technology to quite different
types of products. And this is something that we fundamentally have to
accept and actually, in a certain sense, be maybe, uh, even grateful for because Google is
providing something to the world that we all rely on day to day. The reliability ultimately, and the trust needs to
be upheld in these kinds of, in those offerings and those products. Google search, uh, first and foremost is something you
just have to be extremely careful around. And so I think we've seen in, in recent
launches where Google has started to

add AI capabilities and, you know, deployed the Gemini
models in search. That these things are not actually easy to get right. It's easy to underestimate the amount of additional
care you have to take when you work on products like that compared to, you know, basically trailblazing
into an entirely new category where, certainly, you know, ChatGPT has said many a
weird thing on the way. Did ChatGPT put pressure on companies to go to market
with products like this too soon? I wouldn't say that that's the direction of the
pressure. It certainly has put pressure on companies to put
stuff out in the marketplace, and that's great. That's what propels us forward to soon. I don't think so. We see it as problematic launches
because of the fact that, you know, we've come to rely on these products so
heavily and, you know, in a certain sense, almost blindly. That's why they are jarring to us.

But ultimately, if we want those products to evolve as
fast as they can, we have to live with a bunch of botched launches here and there. That's just par for
the course. Our initial Google Translate launches were so much
worse than any of the stuff that we're seeing here now in terms of just odd output that you could provoke
and, you know, glitches that we had. I mean, in a certain sense, you could argue the
first versions of Google Translate were more glitch than they were product. Right. But it was necessary. It was absolutely necessary to
get us to the point where we're at now. Google translate as a product was, in a certain form,
necessary to exist for not only the transformer to come about, but sequence to
sequence learning with neural networks, so that the modern applications of LSTMs, at least in modeling
language and understanding language, were, you know, at least a good fraction of them were motivated
by machine translation and and done by

people either in the translate team or around the translate team, and so forth. So it's just par for the course. This is its bleeding-edge technology. And sometimes it also looks like that. As one of the originators of the transformer, do you feel a sense of responsibility around making sure that generative AI advances responsibly, securely, and without bias? That is, lumping together a lot of things that I think we need to tease apart, deploying these responsibly in a way that actually maximizes benefit to humanity. Absolutely, absolutely. When it comes to the question of bias, it is not necessarily something that I would attribute at all to the technology. That bias emanates from the data. And yes, certainly you can select data and you can curate it, and that's something you have to do. But I believe thinking that the kind of the core network architecture is at fault if you end up with, you know, a model that overemphasizes certain

biases present in its training data. That's not the smoking gun we're looking for. Now, do I feel strongly that data curation and and
quality control, etc. monitoring needs to happen in general, regardless of
whether it's a transformer or an RNN or God knows what future network architecture people will deploy. Absolutely. And I think these things are are
essential. And I do think that it's actually a research direction
overall, but still underinvested in, um, and underemphasized in terms of, say, for example,
developing tooling for making this easier and teams to react faster to say, for example,
glitches or undesired behaviors of these models. Something worth mentioning is that we need to make
sure that we don't let ourselves get distracted by anxiety or fears of, you know, very
difficult to estimate potentially very long term so-called
existential risks in this space, because we're

probably better off investing at least some of the
time. Actually, on addressing some of the more mundane questions, say, for example, around curating data.
Better to not remove bias, by the way, but to have the bias reflect the biases that we intend. Because just to say this out very clearly in machine
learning, learning is what you learn is, in a certain sense, mathematically speaking, a form of
bias. All models are biased, otherwise they haven't learned
anything. It just has to be the case that you know, we are happy
with the biases that those models later exhibit exhibit, and that's not always
the case. On the flip side, let's talk about what's possible and
what's exciting. And I want to start with a question that we at CNBC
think about a lot, which is how is this going to actually be profitable for companies? Can you name some use cases that will turn a profit? Maybe I'm slightly too biased here, but I actually feel
media.

uh, whether it's text, videos, etc., might not be the easiest, um, space to deploy these
models profitably in the short term. Say, for example, using AI to identify, uh, highly
promising drilling sites for geothermal energy, which is done, which can be done
with generative models that are quite similar to the ones used to generate, you know, funny one-minute
video clips or so using generative AI in ways similar to what we do at Inceptive, where we
use generative AI to design RNA molecules that later form the core of what we see as a
next generation of medicines. There's no question that over the next handful of
years, certainly before the end of this decade, we'll see medicines
hopefully saving many people's lives, probably also having generated
quite some value. And so I feel deploying this, you know, in profitable
ways really is is is guaranteed to happen. Something like medicine is high risk but potential high
reward use case

was something like that possible at Google, or is that what made you leave in 2021? It's definitely possible. Google alphabet overall has interesting efforts in this space. More than one. Calico comes to mind. Verily, there's a whole bunch of others that are not as or not public at all pioneering a technology like this in partnership with large pharmaceuticals, which we believe is actually probably the best way, the most effective way, and the way in which we can get this fastest into patients is something that is much easier to do if you do not have a $2 trillion brand. Explain what Inceptive is. Inceptive is the first biological software company. Now what that actually means, what we think of as biological software are molecules whose exact composition has been designed synthetically, but that ultimately are instances of biological macromolecules such as RNA, DNA, proteins, etc.

we're starting with a focus on RNA, whose exact
composition has been designed with generative artificial intelligence, such that these molecules
inside certain biological systems exhibit behaviors that ultimately are native to those
systems. So there's actually this promise of a flavor of
medicine that is in much greater harmony with living systems than most existing
medicines. What are some really tangible ways that Inceptive is
going to use generative AI to transform medicine if your hopes come true? So if you think about it, one thing that was special
about the mRNA Covid vaccines is that they effectively instructed our cells to
produce a protein in a way that really is exactly mimicking the
way ourselves constantly produce protein to keep us alive. That then triggers the immune system to develop
immunity even before you're exposed to to Covid, to the actual pathogen. And while it's a very simple instance of what we call
biological software, you can think about taking that really to the extreme in order to create
medicines with capabilities

that are far away, more advanced than what we can do today. Medicines that, for example, could exhibit certain behaviors only in certain conditions, identify whether or not the medicine has entered a diseased cell or one that's actually healthy, and if they have entered one that's healthy, they might just decide to disintegrate. Nvidia is an investor. What does that partnership look like? It really is a very classic venture investment relationship. We like their GPUs about as much as everybody else. What do you say to the Doomsdayers, the naysayers? I think it is very clear. We've touched on some of those areas. There's so much positive potential, so much potential to address some of the biggest challenges we are facing as a species: climate change, the next disease outbreak, the next pandemic. Yes, there is potential for dual use. The best way of limiting that danger is really to push the development of this.

technology in responsible ways forward, as fast as we
can. And ultimately, while we need to keep an eye out for
runaway effects. So existential risk, as far as I can tell, we're far,
far away from that. Is generative AI possibly a bubble? I would say it is almost certain that in some areas of
application, generative AI is maybe hyped to an extent that you
could call a bubble by and large. But when you include applications in the
sciences, I think we're underestimating the potential dramatically. What are some ways that generative AI is going to
change companies business models? All sorts of processes requiring complex communication
can be dramatically accelerated and or scaled using generative AI. One example here is, for example, is education, where
generating effective educational content is something that has
certainly been bottlenecked by cost. Where generative AI could drive that cost down,
could enable us to generate content that is actually

optimized for an individual instead of, you know, for
an entire generation if you wish. But there are many other similar applications for
making communication more effective, more scalable, or cost-effective. Ultimately, generative AI will really
change business models because it will take either a major cost factor out of the equation, or allow
far greater reach. Speaking of education, what should students these days
study if they want to have a, you know, tee up a career in this space down the road? Again, I might be a bit biased, but I think really a
combination of natural sciences, a healthy dose of math and, you know, a good helping of computer science. But really, I don't think in the traditional sense
there is a single major that one could focus on or should focus on in order to both be effective
quickly, but also future-proof your education. It seems like it's disrupting just pretty much every
space, but it does cost

billions of dollars to train these models, and more energy than we really even figured out where it's going to come from. Is it worth it? We will find over time that our ability to harness energy, whether it's directly solar hitting our planet, whether it's geothermal, and everything in between. Fusion comes to mind as something that we don't know when it's going to happen, but eventually it hopefully will. AI will enable use of energy and will make our use of energy so much more efficient that really that will completely counterbalance power or energy consumed by AI, even for maybe the more mundane applications like image video generation or understanding language generating texts. Jakob, thank you so much for joining us today. Thank you for having me.

Inceptive CEO Jakob Uszkoreit On Transformers And Using AI To Make New Drugs

Welcome to the official YouTube channel for CNBC. From 'Wall Street' to 'Main Street' to award winning original documentaries and Reality TV series, CNBC has you covered. Experience special sneak peeks of your favorite shows, exclusive video and more.

For real-time news updates from around the world, visit: http://www.cnbc.com

For CNBC's primetime TV lineup, visit http://tv.cnbc.com

For CNBC's International YouTube channel, visit: http://www.youtube.com/cnbcinternational

CNBC

The Original Idea behind Transformers and Its Evolution

Jakob Uszkoreit is credited with the original idea for Transformers, developed to overcome the limitations of existing neural networks' training speed and data usage.

The concept of self-attention, proposed by Uszkoreit and his colleagues at Google in 2017, led to the creation of the transformer architecture.

The monumental impact of the transformer was not fully anticipated during the initial paper's development, as it was seen as one step in a series of advancements in neural network models.

The Impact of Transformers on Google's Products

Transformers have been applied to a variety of problems, from protein structure prediction to computer vision, including Google search, Assistant, and Translate.

The deployment of transformers in Google's products predates the launch of ChatGPT as a consumer product.

Google's cautious approach to implementing transformer technology is attributed to the company's need for reliability and trust in its offerings.

The Challenges of Deploying Generative AI Models

Deploying AI models like Gemini in search is more challenging than pioneering new categories.

ChatGPT has pressured companies to release products quickly, despite the potential for problematic launches.

Despite the glitches, the evolution of AI products is necessary, similar to the early stages of Google Translate.

The responsibility of advancing generative AI technology includes ensuring responsible deployment, security, and addressing bias in the data.

The Importance of Data Curation in Machine Learning

Data curation and quality control are essential in machine learning and require more investment and emphasis.

It is crucial to address biases in training data and ensure that the intended biases are reflected in the models' behavior.

Deploying AI models profitably in the short term may be challenging in the media industry, but use cases in geothermal energy and medicine show guaranteed profitability in the near future.

Inceptive: Pioneering Biological Software for Advanced Medicine

Inceptive is the first biological software company, using generative AI to design molecules that behave in harmony with biological systems, with a focus on RNA.

The company aims to transform medicine by creating advanced medicines that can exhibit specific behaviors, target diseased cells, and disintegrate in healthy cells.

Nvidia's partnership with Inceptive is a classic venture investment relationship, leveraging the use of GPUs.

The company believes in the positive potential of their technology to address major challenges such as climate change and disease outbreaks, despite the potential for dual use.

Generative AI is experiencing hype but has underestimated potential in the sciences.

It will significantly accelerate and scale processes in business, particularly in areas like education, making communication more effective and cost-efficient.

Students interested in a career in generative AI should focus on natural sciences, math, and computer science, as there is no single traditional major to future-proof their education in this rapidly disrupting field.

Despite the high cost and energy consumption of training AI models, the potential for more efficient energy harnessing, driven by AI, is expected to counterbalance its environmental impact in the long run.