There's this area of computer science that is so difficult to write code for that only the largest companies in the world are able to do so, and even then they spend years and millions of dollars with huge teams just trying to solve these problems. It's called automated

reasoning, and the cost of getting something wrong within these systems is catastrophic. But what if we used AI? AI has already changed the face of coding basically overnight. LLMs can write a ton of code, whether they're helping you write code or writing the entire code

for you, but they have one massive problem: hallucinations. And a single hallucination can cause the entire system to fail. So, does that mean we're stuck writing automated reasoning by hand forever? It turns out we may have actually finally solved this problem.

previously took us years to build and really only the largest companies could do it can now be built with just a single person in a few hours? So let me break it all down for you: why is automated reasoning so difficult? Why is it so mission-critical, and why is

failure not an option at all? I'll give you examples, I'll give you a little bit of the history, and by the end you're going to be convinced that we're on the cusp of an automated reasoning revolution. So let's start with the

basics: what is automated reasoning? But before I get into that, I want to give a special thanks to my partner on this video, AWS. Why they wanted to partner with me on this video is going to be clear as I explain more about this topic. So, automated reasoning is the science of

Using mathematical logic to prove the validity of statements. It sounds confusing, but it's really not that confusing at its most basic level. Think about this: when it rains, the ground gets wet. When the ground gets wet, tires have less grip. Thus, when it rains, tires have less grip.

That is called the transitive property, and that example is super simple to understand, but it can actually be proven using mathematics. Automated reasoning needs to be perfect; it has to be perfectly precise; it cannot have any ambiguity because it has to be proven with math. And that's a very simple example, but imagine you have a rule set that is incredibly complex.

Let me give you a few examples. Think about an airline company; they have policies for giving refunds, and these are not easy, simple policies. These are highly complex policies—dozens of pages of rules and situations and what to give, when to give it, how much to give, and if there's even one loophole in that logic to decide how to give or what to give in terms of refunds, somebody will exploit it,

and that will be catastrophic for the airline. Now imagine having to take that rule set and converting it into code. That is incredibly difficult, and I am going to give you another example. AWS has a system called IAM, and it is their permissioning system. It is the system that decides who should have access to what and what level of access they should have.

Now they have dozens of different services they offer; they have essentially unlimited different types of permissions. So with all of that, the cost of giving somebody the wrong set of permissions again is catastrophic.

When I spoke to Amazon, they told me that it took them years to build the IAM reasoning system to prove the correctness of the code that implements the policy interpretation.  Not only that, it cost millions of dollars; they had to hire a team of logicians—not magicians, but logicians—to help them write all of this code.

These are professionals who specialize in the science of logic. They got together with a bunch of computer scientists, a bunch of programmers, and essentially took years to map out all the different types of permissions they might want to

give, and then they coded it all. Again, how many companies in the world do you think are able to do this? It turns out that even massive companies like airlines don't do it themselves; they go to a company like Amazon to solve this problem for them. With automated

reasoning, there's an entire mathematical language and a paired coding language that helps you write this mathematical language to help prove the truthfulness of statements. That code is very difficult to map out and write. Now I go back to our original question: why can't we use

artificial intelligence, large language models, to help us write that code? Well, the reason is hallucinations. Hallucinations are a feature, not a bug, of large language models. It is why large language models are able to do what they're able to do; it is predictive,

it is non-deterministic. But how do we take that non-determinism and implement it in a deterministic way, in a way where we know with confidence 100% of the time it will be right? If we were able to solve that problem, we might

be able to take something that took years and millions of dollars to build and get it down to something that just a single person can do over the course of a few hours. And it turns out AWS has solved it and they just announced automated reasoning as part of their

Bedrock service during the re:Invent conference. So let me give you the gist of how this works: you can take any policy that you have—let's say for a hospital on how to treat patients, or a refund policy for an airline, or even what I'll show you today, the leave of

absence policy for a job. Now this is obviously a very simple example, but imagine how complex it really could become. So let's say you have this leave of absence policy and it is written in English; it's written by the HR department of a company; it has things

like a certain number of days or weeks you could take off based on how long you've been at the company, your seniority, or other characteristics. What AWS has figured out how to do is to be able to load up that natural language document, the policy document, to use AI to

actually create the logical rules for it, extract the different variables, and then convert that all over to a logical language—something that can be proven with 100% certainty. Amazon Bedrock guardrails, automated reasoning checks, is the first and only generative AI safeguard that helps prevent factual errors due to

hallucinations using logically accurate and verifiable reasoning that explains why generative AI responses are correct. Let me show you this example. So we have this leave of absence policy, as I mentioned, it's only two pages long, but it has a lot of information in it. So to

To be eligible for the leave of absence policy, you must have more than 10 years of tenure at the senior level or higher, such as vice president or higher. It shall not exceed 12 weeks in any consecutive 12-month period. So, a lot of rules and a

lot of potentially conflicting rules that actually need to be mathematically proven not to be conflicting. So we're going to take this document and we're going to upload it to Amazon's automated reasoning system. Now, if I pause here for a second, we can see the policy details

at the top, the uploaded document midway through, and then optionally an intent that we're looking to accomplish with our automated reasoning. The ultimate goal is to be able to automate responses to employees who have questions about the leave of absence policy. No longer do they have to read through the policy

themselves or go to HR directly; they can just ask AI. AI might not be able to answer with 100% certainty, but Amazon has the tools to get as close to 100% certainty as possible—a level significantly closer to perfection than

current RAG systems offer. So, once we've uploaded all of this, we go and we click 'Create Policy,' and this is where the magic happens. It is interpreting all of the natural language in that document and coming up with a set of rules that can be proven using logic for that

policy. So what you can see here is the actual rules of the policy extracted from that natural language document. Not only that, but all of the variables that our AI found from that document. So let's look at the top. So we can see here: if an employee is on paid leave of absence,

Then their salary will change upon their return. If an employee is on paid leave of absence, then their sign-on bonus is suspended, and so on. All of these rules have been extracted from that document; it's pretty incredible. Then, at the bottom, they've also

extracted all of the terms or variables. They've given them a type and a description. Pretty insane! But that's not enough; we actually need to be able to test it. We need, as humans, to be able to have confidence in these systems so we can actually ask it questions and

validate whether it's true or false. So let me show you an example. So we're going to ask a question: "I'm a temporary contractor; am I eligible for LOAP?" And the answer we're going to test against is yes: "Temporary employees are allowed to use LOAP." Now, here's the thing that's

not actually true, and what we're seeing at the bottom here is proof of why it's not true. So here are the relevant extracted variables: "Is full-time employee," "LOAP status," and the actual rule from the document: "An employee is eligible for leave of

absence paid if and only if they are full-time, work at least 20 hours per week, and so on." So they've shown explicitly why that statement is not true. So one thing you're going to notice about why this was invalid is because it

made an assumption about the employee level, but there's really no reason to make an assumption. So here it assumed that a temporary employee is an employee level below senior, but that's not necessarily true. So we can actually use natural language to better inform how to

To answer these questions, let me show you. So I'm going to go back to the definitions, I'm going to look for the employee level variable, and I'm going to edit it. I'm going to specify: expect the level to be explicitly mentioned in the question; do not make assumptions.

We have a new change. We're going to save it, and there we have our newly updated policy, which we can test. So there are two different versions. Let's go back to our test playground now and let's ask a similar question: I'm a part-time contractor; am I eligible for LOAP?

Still, the answer is false, but this time it also says 'Is full-time: false' and gives a description of why. Now let me show you the true case if it's a full-time employee. So: I'm an employee; am I eligible? Yes, you are allowed. So it gives the answer, and it is valid.

And so you can configure all this, and it's all done with natural language—that's the best part! No longer do you have to hire teams of engineers, teams of logicians; you just write it yourself.

With the power of AI and the secret sauce that AWS is doing in the background, it does all of the hard stuff for you. Now, automated reasoning is falling within their guardrails feature, and guardrails are used to align LLMs, basically not allowing it to be jailbroken, as best as possible; not allowing it to give certain responses and other alignment measures.

So now you're actually going to be able to ensure the truthfulness of a large language model. This is essentially identifying and squashing unwanted hallucinations—pretty incredible!

Another way this problem has been solved is with contextual grounding. But contextual grounding, although successful, is not explainable. So, similar to RAG, contextual grounding isn't bad; it just really depends on the context in which you use it and your requirements.

So, in previous implementations, you have this source material, for example, that policy document, and you essentially load it up into the large language model and ask it questions about that. But here's the thing: it is still subject to hallucinations. There is nothing mathematically proving that something is

true or false, and that is why automated reasoning is so important for these critical use cases. So that is what AWS just launched at the re:Invent conference: automated reasoning checks in Amazon Bedrock guardrails. This is their sixth type of safeguard being launched into

their guardrails product, and in my mind, this is probably the most important. So check out the AWS news blog for more information. I'll drop all the links in the description below. I want to thank AWS for working with me on this video again because I found this topic to be

absolutely fascinating. So if you enjoyed this video, please consider giving it a like and subscribe, and I'll see you in the next one.

Amazon's NEW AI Just Solved The HARDEST Programming Problem

Artificial Intelligence (AI), Open Source, Generative Art, AI Art, Futurism, ChatGPT, Large Language Models (LLM), Machine Learning, Technology, Coding, Tutorials, AI News, and more

** Exclusive Pinecone Partner **

Matthew Berman

The Challenge of Automated Reasoning and the Promise of AI

Automated reasoning is extremely difficult to code, requiring massive resources and expertise.

AI, specifically LLMs, offers potential for code generation but suffers from hallucinations, leading to system failures.

A potential solution to the challenges of automated reasoning has emerged.

The possibility of building complex systems in hours instead of years is explored.

Understanding Automated Reasoning and its Critical Applications

Automated reasoning uses mathematical logic to prove statement validity, requiring perfect precision and no ambiguity.

Examples include airline refund policies and AWS's IAM permissioning system, where errors have catastrophic consequences.

The complexity of converting complex rule sets into code is highlighted.

AWS's IAM system development involved years of work, millions of dollars, and teams of logicians and computer scientists.

The difficulty of mapping out and writing the code for automated reasoning is emphasized.

Addressing the Limitations of AI in Automated Reasoning

The challenge of using AI (LLMs) lies in their inherent 'hallucinations'—non-deterministic outputs—in a deterministic system.

AWS's solution, announced at re:Invent, enables a single person to build systems previously requiring years and millions of dollars.

AWS's Automated Reasoning Solution: Functionality and Demonstration

AWS's system processes natural language policy documents, extracts rules and variables, and converts them into a logical language.

Amazon Bedrock guardrails, with automated reasoning checks, prevent factual errors caused by AI hallucinations.

A leave of absence policy example demonstrates the system's ability to extract rules and variables from a natural language document.

The system extracts rules and variables, allowing for testing and validation.

The system allows for testing with questions and provides explanations for true/false answers.

The system can be configured using natural language to improve accuracy.

Automated reasoning falls under AWS's guardrails feature, preventing jailbreaking and unwanted responses.

This ensures truthfulness in large language models by identifying and eliminating hallucinations.

Contextual grounding, while successful, lacks explainability, highlighting the importance of automated reasoning.

AWS's automated reasoning checks are the sixth safeguard in their guardrails product.

Amazon's NEW AI Just Solved The HARDEST Programming Problem

스크립트