Panel Discussion: Navigating the AI Landscape - Open or Closed?
Gen AI Summit 2023 - Open or Closed
Zoom: Alex Waibel, Research Fellow at Zoom; Professor, CMU
OpenAI: Boris Power, Member of Technical Staff
Stanford: Percy Liang, Director of Stanford CRFM; Co-founder at Together.xyz
Anthropic: Brian Krausz, Member of Technical Staff
Fellows Fund: Vijay Narayanan, General Partner (Moderator)
Gen AI Summit_ Open or Closed_...AI Landscape - Open or Closed_
Thu, May 25, 2023 11:22AM • 14:04
SUMMARY KEYWORDS
models, companies, ai, people, ecosystem, reproducibility, open source, worked, left, build, api, panel, machine learning, number, data, misuse, safety, source, brian, leveraging
SPEAKERS
Brian Kraus, Percy Liang, Vijay Narayanan, Boris Power
Vijay Narayanan 00:00
about new startups and large number of startups trying to take the other approach of launching things in open source. So standardization AI, dowdy from data bricks, recently invented German models together about coming in. So my company managing now between these two ecosystems and ecosystem for generating AI and those sorts of ecosystem agenda. So the panel today is really about how do we as users. Okay, how do we as users, companies, developers, how do we think about how to navigate this landscape of both the closed and the open source ecosystem? It doesn't have to be necessarily one or the other. It could be a combination of both as well. So with that, I'd like to end the ecosystem. It sort of goes way beyond just the foundation models themselves. We need a lot of tools, frameworks and processes for managing these models for for measuring the values of these models, for understanding the copyright issues, processes and governance of these models. So there's a lot of other tools and frameworks you need. In order to be able to raise these genetic conditions. So this panel today will shed light on. I'm hoping that the panelists will really help us shed light on a number of questions related to this job. With that, I wanted to continue speaking with us, the panelists. It's a very distinguished panel. To my immediate left is Dr. Alex vital. Alex is a faculty computer science faculty at CMU and accounts for the Institute of Technology. He's been a longtime researcher in speech recognition and translation and leveraging networks very extensively in a number of these of these areas. He's also one of the first to my knowledge to invent time delay neural networks. So welcome to life. Balance is Brian Brian. Coach. Ryan is a member of technical staff and throughout the working on productizing and all of the models coming out of control. He has founded two companies before so he has been twice entrepreneur in the past. And he's also worked in Facebook and Stripe comm has a number of other left and right and to the left of Brian is Boris Boris power. So what is Leeds playing a team at open AI, which really is pushing the boundaries of how do you solve real world problems? Using the technologies developed by open air is also an adviser to the open AI fund. He holds an AI degree from the University of Edinburgh and he has worked as a software engineer and data scientists and machine learning engineering. Welcome before and on the far left. We have firstly who needs no introduction. But I just wanted to highlight a couple of things from two journalists introduction of policy this morning was the Athenians verges on robbing edge of reproducibility. So for those of you know, when he in the matter of the coda lab worksheets, which really brought in large scale reproducibility and running a lot of tools for reproducibility and machine learning, specifically for this panel, he's also a co founder of the company together dot XYZ that has the stated mission of making a measurable large scale open source generative AI models as cloud services to do this. So recognizing this panel yesterday, I just wanted to start with a very simple question. We'd love to get a gentleman's thoughts on given this How should the users right so the different companies whether the prices within your startups, large margin small, fun enterprise ones, consumer ones, how should they be really thinking about leveraging what's coming out of both the open and closed source ecosystem for generating the person you want to do? So?
Percy Liang 04:50
Yeah, thank you for an introduction and very happy to be here. Again, I think this is really important and interesting question. And I would say that it's a spectrum and it's actually multi dimensional. So I don't want to think about this as some sort of fundamental autonomy, either. I think my recommendation is that, you know, as has been said, multiple times this field is moving incredibly quickly. What is the best model today might be supplanted by some other model right now and I think if you're building applications downstream, it's important to build them in a bit of a future proof way. And unfortunately, a lot of models aren't interoperable or almost interoperable or behaves a similar way. And I think a crucial thing is to think about these models as each one has a certain cost and has certain performance characteristics. Okay, that this is why I stress benchmark is very important, because themes move quickly as to what might be the best decision might not might not be the best decision later and only through rigorous benchmarking, can you make intelligent decisions whether to switch and you want right now, I think the switching costs to be relatively low because our time and furthermore, I think that we shouldn't think about these models as static. I think one of the benefits of thinking about the open source ecosystem is how rapidly things get adaptive things for example, the JAMA data set that we put out, was I don't remember remember, it was three weeks ago, which feels like eternity. within 24 hours, someone had already been training models on that data set and I looked at hugging face. There's 50 models. Not only that, there's other companies who have mixed that data set with other data sets and are training additional models based on their particular needs. Not to mention once you have these models, all the fine tuning work that can be done. So you think about this kind of whole ecosystem. Where are these assets that have produced and people can and take and mix and match? Much like how software has done. I think software obviously is much more mature than the foundation models ecosystem, but I think that might be a useful metaphor to think about how we manage to make decisions going forward.
Vijay Narayanan 07:42
And Brian would love to hear your thoughts on the companies that are offering those those generated AI. Next step. So how should you think how do you think companies should be thinking about this?
Boris Power 07:56
kickoff? So I think open AI both open source and some models, answers other models, closed source and I think in particular, is open source two different repositories, one, which is automatically goes easier to do, although you've also more complex capabilities, which are not just which company simply assessed by our role based divergence. And the OPlan cookbook, which is reposted, shows the best practices for how to use these models. So I also agree that there is an ecosystem here that we would all love to see development. I think that ecosystem should emulation of those people using open source or closed source models, whatever works best for your use case. But I think one big benefit of closed source models that are served through an API is that they're seeing so many people who previously hadn't done anything in the machine learning space, but they're extremely creative, are able to build these incredible applications and iterate incredibly quickly. So I think, really, what we're seeing is a better grasp of full stack engineer as being a very capable, single person company that can create many different possible product ideas and then see what sticks.
Percy Liang 09:07
And chime in here, I think there's, you know, API's is sort of orthogonal axes. You could have API's for closed or open source models. I totally agree with the power of API's because it makes you don't have to stand up a huge model yourself. But I think that sort of
Brian Kraus 09:29
Yeah, I missed that point. I think of it I'm kind of two different axes. One is the safety axis which has been a lot of my personal focus lately. And I think close soft close source models offer a lot more tools towards safety. You've seen with various safety checks with relatively little effort. And so if we're kind of in the space of quick probing and still trying to wrap our head around it haven't seen mostly built income models for the Model and Model we can sell?
Percy Liang 10:10
A lot of those things that we find
Brian Kraus 10:17
extra queries or various other mechanisms to try and make sense so I think, looking for models that are out there
Boris Power 10:33
just jumping in that band safety thing, cities, a word is used for many purposes. Understanding why companies are doing this, it's this is really not censorship, and it always gets more and more powerful. The dangerous will get higher and higher. It sort of reduces the amount of money or effort somebody needs to close real world harm and it's also development certainly just promote themselves to be dangerous. So scientists naturally as humans choose models to do worse things. And the basic argument that maybe the models that are out there are just for pleasure lines up for a superhuman number of problems, especially in principle binding knowledge for multiple domains that you can hardly find in a single person. So if someone were to distinguish safety from simple content, moderation, and then also with the various capabilities as well.
Percy Liang 11:30
From here, I think maybe the top level is between sort of accidents. There are people who are well intentioned, maybe slightly who wants to do the recommend for those that might assume yes, Robin deploy something. I think it's one of those cases. It's actually you know, open sources. Because Nova community, the majority are good. We should build as a community tools to do basically safety checks and tech bias content. Moderation with the benefit of these these small teams, and auditable and as long as the best practices are established, and it's easy for people to think the second category of misuse is gyms, perhaps I think, a little bit more challenging to deal with. I don't think this is maybe a month unsurmountable I think for misuse, I think there is a you know, a trade off between what you put it out while people can can use it. I mean, I think this is just the nature of dual use technology. Right? With even with internet or social media, there are things you can do, which are good and bad. And then the question is, you know where you control for example, you can make law laws that ban certain types of behaviors regardless of where they're coming from. Night but you are aware, we didn't record
To make sure there's the safeties up to the standards that it can be and serving authority. What can we learn what can be learned these balls and before going to be from all over the phone, how are they even understanding of how powerful these models are? So the only way we think is reasonable to produce progress towards a gradual deployment, where we see slowly what happens and I think the voids of the window or most dangerous and distributed contract. Organizers are from both sides from the side where the pool open too fast and also on the other side of the open source side, which is good after receiving the information. I think it's important to look ahead and I'm the only person say that like we should definitely open source ways which would be five six posts we are well dangerous is that responsibility to do most of the future acceleration risks, because then suddenly, everyone else wants to start changing exactly the same models with exactly the same technology and trying to be faster and out compete and be still competitive dynamics that may result in much less safety overall with these models.
01:14
You got here I find some of the discussion. Because your words few companies now thinking about how they can create a morality that will work for everyone on the planet. That closed doors, and that's just not going to work. The regulator's union in the United States. There is just so much politics that we have answers. So in other chats with the EU and other models already have things where they refuse to answer the second question as to have a president and he says, of course this kind of this was hand programmed, right? I think we need to think about having tradable moralities because different parts of the world will think differently and your attitude towards food was mentioned or different things both in Europe and China and dealing with these issues. Inside one company was a mistake in social media. I think it's going to happen again. In general opinion. So I think we really need to worry about how to formulate the problem of morality in a way that becomes specifiable will curry and trainable locally. So that people can create healthy ecosystems provide guidelines and they want to ensure to ensure the environment so, I think it's cool because we need to discuss some of the ways in which we can impose restrictions like you already do like your message. Oh, he asked about Trump and Biden just don't answer. We can do this directly. Like you can have interactive ways in which like which modifications to provide surface. If you're touching on things that are taboo question on Taiwan short, or the question of Ukraine and united country points, depending on where you're at this plan needs to be carefully in the language to respond to it. And so I think we need to be able to get what you realize is a local language because the language was only as good as provide color and the printing press he talks about measuring the quality and the quality is sometimes subjective. And so we need to have a mechanism of what you're writing such that you have specificity and locality on in different places. So that can be modularized.
04:03
Yeah, I think you're absolutely right. And I think the default world is companies trying to become partners in truth and utter disaster. Yeah, hopefully. Do this. Yeah, I do think that the progress towards that is certainly the same. While there's a lot of research going on, I just think until we have the ability to do that the same way and start to realize that there's meetings this place just want to sit. But if you follow us as long as you have to.
04:38
There's a lot of misconception that internet programming does not have your back end catches from or by winning, but something special with it comes from the fears and dangers around the political and the misinformation today that it's no good for you to be a VR, which is I think, we believe that the balance should be set by what the user wants usually will decide what their preferences are. But we still believe there should be many, many roles as far as possible societal bound. So what's the level of acceptability then given that user should be able to specify what's their worldview? What are their bodies and cognitive strategies and
05:22
you know, where does that come from? I know, it's not a regular expression of the data of how you answer this question. And either because of some sort of constitutional issue, or user feedback, and there's this sort of general sense that walls are aligning to whose values are so you know, can you say actually a little bit more about how that is? If you ask Trump, how does where does that decision came from? Yeah,
05:54
let me just tell you, this is such a trivial example. Right? Because your question can be much more complex and with no end to morality is even the hierarchy of complex issues right. There was this wonderful. Interviewer, Gizmodo on this chat GPT flying on because of an insane he was doing the Blackwater this week. During the captures, right, you know, the story went viral, because changing people's lives and pretending that it was better but it was a blind person trying to catch us. And then the interviewer suggested, oh, I would never lie to be a blind person just to solve the CAPTCHA. You think about this hierarchy of reality that it shouldn't take for your survival. But it's not. It's not just Biden, and Trump. It's just a whole hierarchy inside behind it, which
07:03
is designators for 25 minutes. So, we have in this chair, very faithfully what is it going to be done with a lot of challenges? What do you think
07:18
of the challenges, the challenges and the condition?
07:23
So let me just make a comment. If it's actually following all of this, I think we need to find a mechanism of making tradable or guidelines. You have more automatic and local and local
07:43
challenges. I think safety is always a huge challenge. And I think this ties into some morality, to say this AI is one that refuses to answer questions, but that's also not helpful. So we're not going to turn that and so you know,
07:58
they would make it a directive or ask a question.
08:01
Yes. So there's a lot of there's a lot of nuance in what it means to be safe, and how you can see and I think, one that is built into models and to really figure out how to operate that. You know, a safe model of land is not necessarily safe all through the world and people do crazy things out there. And so figuring out how to layer on much more pleasurable right as you as you said, how to layer on safety, both built into models and wrapped in an ecosystem and in
08:32
terms of watermarking challenging clipping and whatnot. Now we're going to talk about you want to know
08:45
a problem with the traceability of
08:47
Yes, potentially, I think they're ethical questions. I think it's a very good idea, but
08:58
how do we create incentives so that there's no race to the bottom with regards to safety, whatever I'm hungry for research labs, different countries, and nationally, probably established Jarman's values to making these models and also what we want what we think is a reasonable risk to go ahead and record the challenges in our beds as well.
09:26
I'm personally interested in thinking about a different paradigm for how this could work, in the sense that right now the paradigm is we have organizations which gather data from the internet, maybe supplemented with annotations, train a lot of model whether the clothes are open and deployed to users. Right. This is the law that we're used to. But this doesn't have to be. I think we can keep everyone loves for sure. And I think that this is actually a really good example of something that really shouldn't exist. Like why did why does it work? You could think that, you know, I guess, mechanical and all these were the kind of right ways more centralized ways of creating knowledge and videos, do some crazy experiment with with decentralization, and think about it, this is the world's knowledge base, but anyone can go and edit it and change it. Of course, you build safety on top of this ecosystems right here. You can't just have a complete freefall. There's strict governance, but fundamentally, it is an open ecosystem. And it's been remarkably successful. And stable. I actually don't I'm surprised that it's so so good despite this, this fear that you would have 20 years ago and maybe you still have that anyone who's really just going down allies and screw up and then we would have tons of disinformation. So now, Ellen's are different. So I'm not saying that we're just gonna we're gonna be damned, we're done. But I think it's important to remember that this is very early stage, we just discover fire or fire leprosy or whatever, whatever you have. What your favorite analogy is, is we have existence proof that sort of thing. Abilities are are possible. And that we think is about how do we want this technology to develop in society? How do we want who How should people contribute to it? How should the profits and and value is shared across people and how do you keep it safe? And I think these are vastly open questions. And I think we are in a local optima, where I think it's fundamentally yeah, there's this trade off between open and close and safety and transparency, which maybe if we just step back and kind of redesign the system, you know, we will be in a different spot. And I want to emphasize that, again, it's not this. You think of choosing between two things. I think it's a false dichotomy. And I don't know how to do this. I mean, this is what I think about from you know, this is what academia I think, should be thinking about the kind of the next wave of how you architect these socio technical systems. And but I think, you know, this is this is going to be a big deal and I want to make sure we should all make sure that this technology is is adopted. And the ultimate way that benefits us off.
12:38
Thank you personally, personally, because
12:40
of time, I think we will have to wrap up and now let's do whatever whatever last I
12:45
think, just wanted to second that and maybe follow up with the thing I think we need to create a research on plausibility. And confidence metrics, rather just performance because we can never get performance up to that effect the same as the United speech recognition. There's the same problem or language translation is always a person cookie that lasts 1% That hurts, but if we knew what 1% was, you know you can work around that. It's wonderful. Say
13:16
thank you, gentlemen, for a very enlightening panel today and definitely wrapped up so thank you
13:31
all guys, so that conversation probably would have gone on a lot longer. Unfortunately, we do have to keep going here. So we have one more session before we break for lunch. So hang tight, but I'm very, very excited to introduce the next speaker wrong and VP of Engineering at HubSpot leading the conversation on AI solutions for enterprise.
14:09
Great to see everyone here and it's super excited to be like talking about the AI solution for hotspots. And today, I would love to really just take an opportunity to give all of you a little bit more information about what I'm doing and hospitals and so before we do that, and we kick it off by giving us some initial informations about us. And our company is actually based in Boston, I hope, isn't it visit every possible and our focus is trying to view a meeting CRM platform that empowers the customers to manage their entire life's mission. We want to help millions of organizations to grow better. Yes, sir. We have seen tremendous growth over the last couple days into number six, we first released our first VR application products online with the sole focus on others. But over time, we have grown that coverage to subscribers services calm and more recently is CRM apps and operations. So now that we have a new device CRM platforms to cover the entire CRM use cases on every single lifecycles Sam is also under the leadership of two co founders, Brian's and the boss, and our CEOs dominate was the audience today, and three panels by the way. And then we hit a major milestone of meetings a ours two years ago, but one of the case customers and this year's we're talking business into peanuts we're extremely happy about this world, not just because we're getting more hours, but we are very happy that it's a recombinations our customers. There's more and more SMB customers are benefiting from the platform. This extra investment. If you'll take a poll, and you can do this, what's our total addressable market in the next five years? We're going to see the increase from 45 buildings to 72 buildings in the next five years. I hope I convinced you this is a huge market is not only a huge market, it's a huge underserved markets right now. But if I can show you an evolution of technology that can really help to bring this space first. And that's why I'm not talking about Wi Fi in this case. But before doing that. I want to look at right the rise of this SMP company which is actually uncomfortable. If you think about 20 years ago, actually it was related to a major paradigm shift that we have experienced all of us. First is the internet plus shift in your classes. And then comes the mobile plus social media shift. You cannot attend this convention really fundamentally change the way people think and also that the person you're talking to Google, Facebook, Amazon, your name what they actually changed here is that there's a new generations on the news and how they attention are the chains that become much more mechanical half, seven. So and also they're becoming much more motivated to build a business. So now that 77% of SMBs are going to miss out on media to communicate with their customers. And also there's way more SNPs in training Frankie says that they they think software is a correlation. It's a huge increase that occurs. As a result, we're seeing SMB was increasing this aspect to your recent years, and this is a much higher increase as prospects. Most universities models here, it just means there's no problem. security problems. Many of the small and medium business they play, they are impacted by crisis. of this country's is a disconnected solutions. So to just give you a very concrete numbers, in our most recent survey, it's only 22% of the company says that they have excellent data collection solutions and they are looking at many of them the rest of the 78% are looking for a better opportunity coming up. And that's why we see these opportunities as one decided to change the strategies from one single app to a unified platforms. And we're seeing tremendous success from this strategies because they actually saw a customer really want to talk about this is what we want to talk about the next paradigm shift. Of course, the main topic of this session is AI. I read the one thing about AI as a technology. I want to think about AI as a tool to fix that a customer solve customer challenges. Take a step back and say really bad it's most often. So it's not just like the chatty review reviews. It's actually a more than 50 of safety all times and that was meant to ramp up and downs in the last like 50 years here. I'm just going to show you how to do key events, I believe, is changing the current default AI in about 30 years. So first one looks like in 70s that I just myself. And then this is my first major mainstream use cases that actually shows that AI tend to be with humans. And interestingly Monterey Campbell was in the pictures. He was my first manager back in that year research. I learned monster truck games during that age. So after that, well, here comes the accident, and we officially entered the aircraft in 2016 AlphaGo AlphaGo best humans, arguably the most complicated games where company informations so go very good here, but they made and that's of course nobody's really subject to ask. I think this time is a little bit different if you're looking at a lot of work on that point, it depends we're seeing here. They are very domain specific industry, visuals, or chess games. But this time, they actually show well on democratizing AI support every single force to solve everyday task. We've got very simple and that's why I believe this is a paradigm shift is going to have a much major impact on the US. Air Force here comes two explanations of the AI landscape. Afterwards itself many many star coming up. And also all the great company we have nowadays have been thinking about integrating J Gen generated VR products. I don't know how many of you actually have that period. So you may feel like we wake up every single day feel like those anxious but also excited because we actually don't know what's going to happen today. Maybe there's like 10 more people coming up. Or maybe there's 10 more product release coming up. But whenever it's time to have that moment, take a step back to really think about what's the customer challenges that you're facing, and then use that angles to really think about what AI can help to fix that customer challenges.
22:21
If you think about the data, five years of a CRM solutions, you can pre populate this plate it's a five view of prefix, data, insights and actions. We start with our customer base, using the customer data generating insights and then pulling insights. We conduct some actions and whether the actions whichever data you can get executed well is arrived at. You're going to have a very healthy cycles of navigating data and generating customer information from AI as a call pieces. Underneath that, you can think about this execution of attributes that capture the practice. About that is the goal, the customer experience and trying to execute the lesson. So this is the broader customer data five views that we can conditions. The rest of the opportunities. First, plus AI to help us aI have the customer to execute towards. Second, you actually have a huge amount of human efforts. That was happening with me up if you have to use AI to simulate life. Can you see any repeatable tasks if you have used AI to summarize, to predict. So that's why you put a framework into SMBs you can actually pick out three different foundations on what kind of things that we can do. So first, Small and Medium Business School and data science. So that's why they really suffer from like generating insights requires extracting segments, simplified actions, because small and medium is really find it increasingly difficult to actually figure out how to use SAS products with growing several functionalities and topics in in the hospital. There's a lot more system they are using right now. Can we actually use the echo system into one single world and into one single room? I think there are three different directions that we can potentially locate and I'm going to show you like what kind of use cases that we have in HubSpot. I tried to facilitate and the first one about extracting insights. I probably want to go back and talk about some of the cases people traditionally generate. But I would say before 2019 This is probably the most majority of use cases. And obviously we can do it using the previous data to summarize have happened and use the patterns to predict what's going to happen. So here's one of the use cases on the called Predictive lead scoring. And it's actually the first new AI use cases that HubSpot.
25:40
What is kind of impressive given a customers or potential we're trying to predict how likely this person is unless you have that information. The salespersons can use that as a predictions,
26:00
resource and use that as salespeople to progress on people that are most likely to cause headaches. And it's one of the most asked features of our prospects here. So can you ask a second data anomaly connections. We have a lot of data about 1000 colleges. And at any moments, some of this project goes wrong and some of the workflow can go wrong. Can we automatically detect the things that goes wrong and alert users? What's the problem? Did it ask them to place? I have just two use cases. self explanatory to understand what's happening behind the scenes. But what I really want to talk about here is not about how we implement but I want to talk about what we've learned from customers. It's very interesting. Whenever you should these are the predicted features to our customers. expectation management is very important. If you just tell them a numbers, they will have a really high expectation on the numbers. They would think you have to divide your assets in order in order to make it work. If you are not paying them 100 sets, they may feel your system is broken. But one thing's certain things beyond the numbers. They actually want to close its actions. They want to care less about the number itself. They care more about what kinds of actions especially what's the best solutions they can in order to execute in order to solve the problems. So and that's pieces a lot of people ignore everything. Ai problem is just about okay, I'm predicting just number and send it to the customer and that's it. That's the end game is they have a lot of foresight. And then naturally of course the second thing I want to talk about is about actions. So in fact, a lot of SMP customers, they are really suffering from the action itself. And AI can actually play pick roles on this. And I think this is where you have a lot of confidence. So I'm probably not going to spend too much time to explain what's going on behind the scenes in this market. So I'm going to show you a most recent example of the
28:27
start this market actually works this is no.
28:50
Fast
28:59
food baby Wow. This is really good not to share in the flesh. They are our top 10 content in one place.
29:13
So this product is called Content system. So the idea is that we allow people to enter a few problems and then we use the check ups for you reactions. violins are supposed to generate the more concrete answers based on it serve a lot of use cases here so I'm not going to repeat it but you can probably get it so you can optimally generate better outcomes. Allow people to generate better marketing campaigns in hospital help people to reformat their campaigns based on a lot of variables. So here's a some number, reduce the cost 89% of the marketers they use generally I find this informative and also save a lot of hearts. So on average monitor user generated AI, save after three hours per piece of content. And this is a huge productivity by using this type of tools. So that's why we haven't seen a lot of great momentum coming up on this precious. Last a lot of these connected systems. So as we know, as like leveraging different systems and trying to drive to the lowest possible and what can we use AI to facilitate? And here we'll talk about a new product we also released called chatbots. And if you think about Chatbot is the combination in chat activities and HubSpot and we're basically using the knowledge base system Q allows you to into the hotspot features, very good performance and potential recent
31:06
work and you just get the outcome we were
31:09
before we can kind of skip to the end
31:13
of the time. Let's see how you can do chutzpah.
31:26
Pretty good inhale wow this is so easy. Wow. chatbot did we just become best friends? Gosh, this is a teacher.
31:48
And this is our founder that who Ida and developers in higher logics. It's very impressive, so but if you haven't used Caspar yet, we encourage you to go to black chatbot AI and you can think about that as a system. Let's say you have a lot. So, again, a lot of use cases you can try. So one of my most favorite use cases that you can use chatbot for gender. It's actually a very nice business intelligence. is coming out. So some people would ask okay, why this transfer can be better than that champion because they are on our system. So I'll answer experiences. A chatbot can get access to different business comments and also get access to people and to be honest on a very high level, I believe this will become a long term competitive advantage. Chapter is not the model itself. The model is self promoting marketers. It's really about it's really about who has access to a third party business. So yeah, just showing you example, what's the answers? At this quick glance of what kind of keywords that Hassan was ranking for and you can see on the left hand side is the chapter with the answers and on the right hand side. chatbots answered and was much more structured and once you have the access to and this for me I would love to look into style ideas and directions. I think this is going to cause a lot more AI is using the half Bible for sure I just don't have time to cover. It's even a 20 minute conversation here. So I really hope you use this lesson or thinks we can do here. And AI is going to be a paradigm shift loss. And finally, a few numbers we also find very interesting so 80% of s&p business sales and marketing services, believe AI can make it very easy to compete against large business in the industry. And a lot of people really think highly of AI. I think the walk is very interesting. Of course I really want to end the conversation on this. So AI is a happy customer follow its focus on the admins focus. If I think about AI, it's always been developed by the problem you're trying to solve, and then find the right candidates for basic. So I think that's always the best way to fix a problem and to look at how AI can cost you your business.
