트랜스포머: 제너레이티브 AI의 기원과 미래

### 트랜스포머의 기원과 제너레이티브 AI의 미래: 야콥 우스코라이트와의 대담
* 🧠 트랜스포머의 최초 아이디어를 낸 핵심 인물인 **야콥 우스코라이트가 기존 신경망의 한계를 극복하기 위해 이 개념**을 개발하게 된 과정을 설명합니다.
* 💡 2017년 트랜스포머에 관한 논문은 개발 과정에서 이미 다양한 영역에서 **그 잠재력의 초기 징후**가 드러난 일련의 발전 과정의 한 단계로 여겨졌습니다.
* 🤖 우즈코라이트는 실제 제품에 AI 모델을 배포할 때의 어려움과 이점에 대해 설명하며 **책임감 있는 구현의 중요성**을 강조합니다.
* 💊 이제 이야기는 우즈코라이트의 현재 벤처기업인 **인셉티브(생성 AI를 활용하여 의약품용 RNA 분자를 설계하는 생명공학 스타트업)**로 옮겨갑니다.
* 💰 우즈코라이트는 에너지 탐사 및 바이오 제약과 같은 산업에서 **제너레이티브 AI를 수익성 있게 적용**하여 상당한 가치를 창출할 수 있는 잠재력을 강조합니다.
* 🔮 그는 **데이터 큐레이션과 책임감 있는 배포의 중요성**을 강조하면서 AI 시스템의 편향성에 대한 우려에 대해 언급합니다.
* 🎓 우즈코라이트는 미래 AI 분야에서 일하기 위한 교육에 대한 질문에 **특정 전공보다는 자연과학, 수학, 컴퓨터 과학을 혼합하여 공부하는 것**을 추천합니다.
* 🌐 우즈코라이트는 AI 훈련의 에너지 소비에도 불구하고 **향후 에너지 기술의 발전이 전반적인 영향을 긍정적으로 만들 것**이라고 믿습니다.

트랜스포머 개발의 핵심 인물인 야콥 우스코라이트가 트랜스포머의 기원과 영향, 그리고 제너레이티브 AI의 책임 있는 배포에 대해 이야기합니다.

Our guest today, Jakob Uszkoreit, is often credited
with the original idea for Transformers: the foundational concept that makes generative AI
possible. At Google in 2017, he co-wrote a paper about the idea
with seven other colleagues. Then in 2021, after more than a decade at Google, he
left to start Inceptive, a biotech and AI startup with backing from Nvidia. Here's our conversation. Jakob, thanks for joining us today. Thank you for having me. Absolutely. I have heard that the original idea for
Transformers started with you. Is that true? And where did it come from? Well, let's see. In science, I would say there are rarely, if ever, truly original ideas. It's you're always standing on shoulders of giants. But really what happened around that time is that my
group was faced with an interesting conundrum where we had fairly advanced
and expressive

deep neural networks called LSTMs, long short-term
memory networks, in a configuration that was dubbed Seek-to-seek, also
developed at Google around 2014-2015, that ultimately were really showing state-of-the-art results in neural machine translation or in machine translation overall. Yet complexity, their training speed was so slow that
if you had enormous amounts of training data for a given
problem, you would never actually be able to train even on a fraction of the training data that you had
available. And as a result, if you use those types of neural
networks that were in principle more powerful and more expressive than much simpler, say, feedforward
architectures, you couldn't train on as much data, and as a result, would remain inferior to
much simpler networks trained on a larger fraction of the available data. And so, inspired by that problem, really, we looked at
existing mechanisms or architectures that could help us
overcome this limitation where you would both end up

with something similarly expressive to these RNNs or
LSTMs, while actually massively accelerating training times and training
speed on the kind of accelerator compute that we had at the time. That is still quite strikingly
similar to the kinds of accelerators that we have today. And so that led to, and this is maybe an idea
that I contributed here, this idea that we could use a mechanism called attention
mechanism that had been published and proposed in the two years before we
actually started working on the transformer, maybe even three years before that. Use
such an attention mechanism, not to have one signal informed by information in another,
but actually use it in a way that we then called self-attention, where you basically
have a given structure, a signal, say, a sentence or so, attend to itself in order to further
the interpretation or understanding or ultimately vector
space representation that you have of it. And that mechanism is one that fits exceptionally well
onto GPUs.

And so that's then ultimately, maybe if you wish, one original idea among many, many others that gave rise to what eventually ended up being called the transformer architecture, that we then published in 2017. When you were working on the paper, did you have any idea that it was as monumental as we now see it to be? It really was one step in a long sequence of steps on this road. My group had published a paper just a year prior to that, on a model called the Decomposable Attention Model that actually, by and large, was structurally quite similar, but not nearly implemented as well, and not nearly as fast as the transformer. So really, I think we're making a mistake if we look at it as monumental. But then at the same time, we thought if we could really strike this balance effectively between exploiting the compute resources that we had or maximizing the expressive power in a neural network.

architecture that you can support with that flavor of
compute, then that could affect not only, you know, the classic or typical
applications of these neural networks at the time, but open up many, many new ones. But there is a sentence that we've obsessed over for
quite a while about how excited we are to apply this to a variety of other problems, and also other types of
problems going forward, and that we have pretty high hopes for that. Truth is, at the time, we had already started doing
that and had preliminary results and or at least hints of results that would suggest that, yes, indeed, you
can apply this to a broad variety of problems, ranging from things like protein structure prediction all the
way to computer vision. The consumer-facing ChatGPT Revolution is one side
of the coin. But were you already implementing these things at
Google, Google search, Google Assistant, Google translate, those were teams
you all worked on, right?

Yep, that's right. We had some prototypes. Uh, that would apply Transformers in, you know, even
in the very early days. In some of these products, it took a very long time for those to really
reach the light of day. ChatGPT hasn't actually been around as a consumer
product for that long. And so there are actually applications, for example,
at Google and other places where transformers have been deployed in production long before, but also to much,
much less fanfare. Really, it was ChatGPT that shone the spotlight on
these applications, because there's not that much more to the product than really a slick interface
around a very, very well-trained transformer trained on large amounts of data with
lots of compute and of course, engineered super. Well. What is your reaction to the perception that Google had
this wonderful idea internally and let it slip away and has fallen behind? I mean, ultimately Google has grown into, you know, a
large company, and they're not

necessarily notorious for being particularly nimble at
the same time. It's not actually the case that it really took Google that much longer to launch
this in certain products. I feel what we're actually seeing is that
maybe Google here hasn't been able to be as daring as, you know, a much, much
smaller companies such as OpenAI when it comes to applying this technology to quite different
types of products. And this is something that we fundamentally have to
accept and actually, in a certain sense, be maybe, uh, even grateful for because Google is
providing something to the world that we all rely on day to day. The reliability ultimately, and the trust needs to
be upheld in these kinds of, in those offerings and those products. Google search, uh, first and foremost is something you
just have to be extremely careful around. And so I think we've seen in, in recent
launches where Google has started to

add AI capabilities and, you know, deployed the Gemini
models in search. That these things are not actually easy to get right. It's easy to underestimate the amount of additional
care you have to take when you work on products like that compared to, you know, basically trailblazing
into an entirely new category where, certainly, you know, ChatGPT has said many a
weird thing on the way. Did ChatGPT put pressure on companies to go to market
with products like this too soon? I wouldn't say that that's the direction of the
pressure. It certainly has put pressure on companies to put
stuff out in the marketplace, and that's great. That's what propels us forward to soon. I don't think so. We see it as problematic launches
because of the fact that, you know, we've come to rely on these products so
heavily and, you know, in a certain sense, almost blindly. That's why they are jarring to us.

But ultimately, if we want those products to evolve as
fast as they can, we have to live with a bunch of botched launches here and there. That's just par for
the course. Our initial Google Translate launches were so much
worse than any of the stuff that we're seeing here now in terms of just odd output that you could provoke
and, you know, glitches that we had. I mean, in a certain sense, you could argue the
first versions of Google Translate were more glitch than they were product. Right. But it was necessary. It was absolutely necessary to
get us to the point where we're at now. Google translate as a product was, in a certain form,
necessary to exist for not only the transformer to come about, but sequence to
sequence learning with neural networks, so that the modern applications of LSTMs, at least in modeling
language and understanding language, were, you know, at least a good fraction of them were motivated
by machine translation and and done by

people either in the translate team or around the translate team, and so forth. So it's just par for the course. This is its bleeding-edge technology. And sometimes it also looks like that. As one of the originators of the transformer, do you feel a sense of responsibility around making sure that generative AI advances responsibly, securely, and without bias? That is, lumping together a lot of things that I think we need to tease apart, deploying these responsibly in a way that actually maximizes benefit to humanity. Absolutely, absolutely. When it comes to the question of bias, it is not necessarily something that I would attribute at all to the technology. That bias emanates from the data. And yes, certainly you can select data and you can curate it, and that's something you have to do. But I believe thinking that the kind of the core network architecture is at fault if you end up with, you know, a model that overemphasizes certain

biases present in its training data. That's not the smoking gun we're looking for. Now, do I feel strongly that data curation and and
quality control, etc. monitoring needs to happen in general, regardless of
whether it's a transformer or an RNN or God knows what future network architecture people will deploy. Absolutely. And I think these things are are
essential. And I do think that it's actually a research direction
overall, but still underinvested in, um, and underemphasized in terms of, say, for example,
developing tooling for making this easier and teams to react faster to say, for example,
glitches or undesired behaviors of these models. Something worth mentioning is that we need to make
sure that we don't let ourselves get distracted by anxiety or fears of, you know, very
difficult to estimate potentially very long term so-called
existential risks in this space, because we're

probably better off investing at least some of the
time. Actually, on addressing some of the more mundane questions, say, for example, around curating data.
Better to not remove bias, by the way, but to have the bias reflect the biases that we intend. Because just to say this out very clearly in machine
learning, learning is what you learn is, in a certain sense, mathematically speaking, a form of
bias. All models are biased, otherwise they haven't learned
anything. It just has to be the case that you know, we are happy
with the biases that those models later exhibit exhibit, and that's not always
the case. On the flip side, let's talk about what's possible and
what's exciting. And I want to start with a question that we at CNBC
think about a lot, which is how is this going to actually be profitable for companies? Can you name some use cases that will turn a profit? Maybe I'm slightly too biased here, but I actually feel
media.

uh, whether it's text, videos, etc., might not be the easiest, um, space to deploy these
models profitably in the short term. Say, for example, using AI to identify, uh, highly
promising drilling sites for geothermal energy, which is done, which can be done
with generative models that are quite similar to the ones used to generate, you know, funny one-minute
video clips or so using generative AI in ways similar to what we do at Inceptive, where we
use generative AI to design RNA molecules that later form the core of what we see as a
next generation of medicines. There's no question that over the next handful of
years, certainly before the end of this decade, we'll see medicines
hopefully saving many people's lives, probably also having generated
quite some value. And so I feel deploying this, you know, in profitable
ways really is is is guaranteed to happen. Something like medicine is high risk but potential high
reward use case

was something like that possible at Google, or is that what made you leave in 2021? It's definitely possible. Google alphabet overall has interesting efforts in this space. More than one. Calico comes to mind. Verily, there's a whole bunch of others that are not as or not public at all pioneering a technology like this in partnership with large pharmaceuticals, which we believe is actually probably the best way, the most effective way, and the way in which we can get this fastest into patients is something that is much easier to do if you do not have a $2 trillion brand. Explain what Inceptive is. Inceptive is the first biological software company. Now what that actually means, what we think of as biological software are molecules whose exact composition has been designed synthetically, but that ultimately are instances of biological macromolecules such as RNA, DNA, proteins, etc.

we're starting with a focus on RNA, whose exact
composition has been designed with generative artificial intelligence, such that these molecules
inside certain biological systems exhibit behaviors that ultimately are native to those
systems. So there's actually this promise of a flavor of
medicine that is in much greater harmony with living systems than most existing
medicines. What are some really tangible ways that Inceptive is
going to use generative AI to transform medicine if your hopes come true? So if you think about it, one thing that was special
about the mRNA Covid vaccines is that they effectively instructed our cells to
produce a protein in a way that really is exactly mimicking the
way ourselves constantly produce protein to keep us alive. That then triggers the immune system to develop
immunity even before you're exposed to to Covid, to the actual pathogen. And while it's a very simple instance of what we call
biological software, you can think about taking that really to the extreme in order to create
medicines with capabilities

that are far away, more advanced than what we can do today. Medicines that, for example, could exhibit certain behaviors only in certain conditions, identify whether or not the medicine has entered a diseased cell or one that's actually healthy, and if they have entered one that's healthy, they might just decide to disintegrate. Nvidia is an investor. What does that partnership look like? It really is a very classic venture investment relationship. We like their GPUs about as much as everybody else. What do you say to the Doomsdayers, the naysayers? I think it is very clear. We've touched on some of those areas. There's so much positive potential, so much potential to address some of the biggest challenges we are facing as a species: climate change, the next disease outbreak, the next pandemic. Yes, there is potential for dual use. The best way of limiting that danger is really to push the development of this.

technology in responsible ways forward, as fast as we
can. And ultimately, while we need to keep an eye out for
runaway effects. So existential risk, as far as I can tell, we're far,
far away from that. Is generative AI possibly a bubble? I would say it is almost certain that in some areas of
application, generative AI is maybe hyped to an extent that you
could call a bubble by and large. But when you include applications in the
sciences, I think we're underestimating the potential dramatically. What are some ways that generative AI is going to
change companies business models? All sorts of processes requiring complex communication
can be dramatically accelerated and or scaled using generative AI. One example here is, for example, is education, where
generating effective educational content is something that has
certainly been bottlenecked by cost. Where generative AI could drive that cost down,
could enable us to generate content that is actually

optimized for an individual instead of, you know, for
an entire generation if you wish. But there are many other similar applications for
making communication more effective, more scalable, or cost-effective. Ultimately, generative AI will really
change business models because it will take either a major cost factor out of the equation, or allow
far greater reach. Speaking of education, what should students these days
study if they want to have a, you know, tee up a career in this space down the road? Again, I might be a bit biased, but I think really a
combination of natural sciences, a healthy dose of math and, you know, a good helping of computer science. But really, I don't think in the traditional sense
there is a single major that one could focus on or should focus on in order to both be effective
quickly, but also future-proof your education. It seems like it's disrupting just pretty much every
space, but it does cost

billions of dollars to train these models, and more energy than we really even figured out where it's going to come from. Is it worth it? We will find over time that our ability to harness energy, whether it's directly solar hitting our planet, whether it's geothermal, and everything in between. Fusion comes to mind as something that we don't know when it's going to happen, but eventually it hopefully will. AI will enable use of energy and will make our use of energy so much more efficient that really that will completely counterbalance power or energy consumed by AI, even for maybe the more mundane applications like image video generation or understanding language generating texts. Jakob, thank you so much for joining us today. Thank you for having me.

Inceptive CEO Jakob Uszkoreit On Transformers And Using AI To Make New Drugs

Welcome to the official YouTube channel for CNBC. From 'Wall Street' to 'Main Street' to award winning original documentaries and Reality TV series, CNBC has you covered. Experience special sneak peeks of your favorite shows, exclusive video and more.

For real-time news updates from around the world, visit: http://www.cnbc.com

For CNBC's primetime TV lineup, visit http://tv.cnbc.com

For CNBC's International YouTube channel, visit: http://www.youtube.com/cnbcinternational

CNBC

트랜스포머의 독창적인 아이디어와 그 진화

기존 신경망의 학습 속도와 데이터 사용량의 한계를 극복하기 위해 개발된 트랜스포머의 독창적인 아이디어는 Jakob Uszkoreit의 공로로 인정받고 있습니다.

2017년 우슈코라이트와 그의 동료들이 구글에서 제안한 자기 주의라는 개념이 트랜스포머 아키텍처의 탄생으로 이어졌습니다.

초기 논문 개발 당시에는 트랜스포머가 신경망 모델 발전의 한 단계로 여겨졌기 때문에 그 엄청난 영향력을 완전히 예상하지 못했습니다.

트랜스포머가 Google 제품에 미치는 영향

트랜스포머는 단백질 구조 예측부터 구글 검색, 어시스턴트, 번역을 포함한 컴퓨터 비전에 이르기까지 다양한 문제에 적용되어 왔습니다.

Google 제품에 트랜스포머를 도입한 것은 ChatGPT가 소비자 제품으로 출시되기 이전부터였습니다.

Google이 트랜스포머 기술 구현에 신중하게 접근한 이유는 제품에 대한 안정성과 신뢰가 필요했기 때문입니다.

제너레이티브 AI 모델 배포의 과제

Gemini와 같은 AI 모델을 검색에 배포하는 것은 새로운 카테고리를 개척하는 것보다 더 어려운 일입니다.

ChatGPT는 출시에 문제가 발생할 가능성이 있음에도 불구하고 기업들에게 제품을 빠르게 출시하도록 압력을 가했습니다.

결함에도 불구하고 Google 번역의 초기 단계와 마찬가지로 AI 제품의 진화는 필요합니다.

인공 지능 기술을 발전시키기 위해서는 책임감 있는 배포와 보안을 보장하고 데이터의 편향성을 해결해야 할 책임이 있습니다.

머신 러닝에서 데이터 큐레이션의 중요성

데이터 큐레이션과 품질 관리는 머신러닝에서 필수적이며 더 많은 투자와 강조가 필요합니다.

학습 데이터의 편향성을 해결하고 의도한 편향성이 모델의 동작에 반영되도록 하는 것이 중요합니다.

미디어 업계에서는 단기간에 AI 모델을 수익성 있게 배포하는 것이 어려울 수 있지만, 지열 에너지와 의료 분야의 사용 사례를 보면 가까운 미래에 수익성이 보장되는 것으로 나타났습니다.

창의적: 첨단 의학을 위한 선구적인 생물학 소프트웨어

인셉티브는 최초의 생물학 소프트웨어 회사로, 생성 AI를 사용하여 생물학적 시스템과 조화롭게 작동하는 분자를 설계하며 RNA에 중점을 두고 있습니다.

이 회사는 특정 행동을 보이고, 병든 세포를 표적으로 삼고, 건강한 세포에서 분해될 수 있는 첨단 의약품을 개발하여 의학을 혁신하는 것을 목표로 합니다.

엔비디아와 인셉티브의 파트너십은 GPU를 활용한 전형적인 벤처 투자 관계입니다.

이 회사는 기후 변화 및 질병 발생과 같은 주요 과제를 해결할 수 있는 이 기술의 긍정적인 잠재력을 믿고 있습니다.

It seems there was no text provided for me to work on.

Please share the sentence you'd like me to highlight the key phrase in, and I'll be happy to assist!

제너레이티브 AI는 과학계에서 과대 포장되어 있지만 그 잠재력은 과소평가되고 있습니다.

특히 교육과 같은 분야에서 비즈니스 프로세스를 크게 가속화하고 확장하여 커뮤니케이션을 더욱 효과적이고 비용 효율적으로 만들 수 있습니다.

Please share the sentence you'd like me to highlight, and I'll be happy to assist!

빠르게 변화하는 이 분야에서 미래를 대비할 수 있는 전통적인 전공은 하나도 없기 때문에 제너레이티브 AI 분야에 관심이 있는 학생이라면 자연과학, 수학, 컴퓨터 과학에 집중하는 것이 좋습니다.

Please provide a sentence or paragraph, and I'll help you highlight the key phrase!

AI 모델 학습에 드는 높은 비용과 에너지 소비에도 불구하고, 장기적으로는 AI가 주도하는 보다 효율적인 에너지 활용의 잠재력이 환경에 미치는 영향을 상쇄할 것으로 예상됩니다.