Panel Discussion: Dissecting the Modern AI Stack
Gen AI Summit 2023 - Open or Closed
Zoom: Vijay Parthasarathy, Head of AI
Fireworks.ai: Lin Qiao, Cofounder and CEO, Co-creator of Pytorch at Meta
ServiceNow: Ravi Krishnamurthy, Sr. Director of Product
Google: Jianchang Mao, VP of Engineering
Instacart: Haixun Wang, VP of Engineering (Moderator)
SUMMARY KEYWORDS
models, companies, foundation, cost, problem, tasks, product, extreme, fast, infrastructure, choose, solve, managing, applications, agility, question, invest, innovation, depending, work
SPEAKERS
Haixun Wang, Jianchang Mao, Ravi Krishnamurthy, Lin Qiao, Vijay Parthasarathy
00:00
capacity and non deterministic behavior. So, a family of seeds to provide some mechanism to contribute assure that the usability of a foundation models for that experiment capture module configurations establishing relationships and then this developers to introduce them yeah
Haixun Wang 00:36
definitely these are all great points and there are so many we need this so much you meet with some of the financial models to require us to see this kind of different recovery This is our second question which is, you know, many companies know better companies you have like maybe 10 or 20 people managing the entire mouse that entire companies will already have as an ops which are currently you know, supporting like already currently on the tour 100 traditional maturity model. Predictive maturity models are how they come together. How do you know traditional models the stack would have developed for the for tradition models come together with this you know, you know, very important foundational Bono component right, a lot of company including companies like Databricks need to look cheap, they provide all the all the ML you know, standard for like ML flow for managing the ML lifecycle, but the blown they are also diving into your validation models. So your opinions, especially for maybe a user perspective to sales will eventually will come together
02:13
think it comes down
02:13
to what hasn't changed is the customer problem
Ravi Krishnamurthy 02:17
solving the same process? How do we able to be more productive ServiceNow we're so similarly how do we improve our quality of delivery? While the most fundamental problem how do we manage all this fundamental customer problems? And the question then comes are these new things solving the problem and then I was just leaving Mexico going to these conventions like some of our older discriminative models. They cost me fractions of a cent. And we sometimes tend to no pun intended in this comment so is that something should I replace that with something that costs 10 100 times more? Am I good enough? Or should I not? So I do think this and like Richard was saying, and he was saying, what are the benchmark? How do I know when can I replace so there's a lot of those questions we got to figure out. So I think there is going to be a period of time that exists together and depending on how the cost comes down, a prolonged period so I am again lights on the same I am the next show will rise and assuming that the world exists in two random together and there are some pipeline architectures we have we can combine things like that that we are doing in our analysis. These are all things.
Vijay Parthasarathy 04:03
I took at least two years especially other catalysts. Paladins and the design choice depends on a number of parameters, or factors, whether to choose traditional, specialized model versus 5g Foundation model. Those factors including the code for example, how special your use cases and performance requirements speed accuracy and so on. And the availability of computing resources, that's a big factor in cost per serving. Those are the big factors, but I believe that as foundation model improves, and become more capable, and more and more specialized. Machine Learning models can be replaced by fine tuning foundation models, because the foundation model can scale very easily horizontally across multiple paths. A very diverse set of tasks. It's much easier to scale horizontally than specialized models, as that's the main main benefit. So the development development cost or job and how to lower the computational cost. Currently is a high budget we're willing to do so over time. So I can in the short term, I can think of four scenarios. When is that happening? And the companies can leverage the best of two worlds depends on the design choice the factors and decided which which paradigm to use. The second scenario is use a foundation model as a building block. And use is often used as a building block and to stand upon to fine tune in to customer to your specific applications. The third scenario is maybe internal controllability of unity. And then the first scenario is put a heads up face as foundational model
Lin Qiao 06:47
Okay, that's a good thing, charity if he did I do kind of maybe to make this conversation more interesting. Like we can talk about two extremes, right. So one extreme is, one says lose along. There's one financial model that's gonna dominate all entire audit tasks, and we don't need anything else. And the other extreme is we have podcast specific smaller models. And there could be and we have user pick and choose your business enterprise kind of pick and choose whichever is the best one and the Compose to solve their business problem. So like if we look at these two extreme spectrum of possibilities, I think we can analogy with the US is the consumer market, right? How many cars do you have in your house so yeah, so if you think about cars, right? You can buy a truck, it can do a lot of heavy lifting, and you can use it to do many things, but it's very inefficient. If you're really efficient. You can have your small commuter cars and you drive between your home in the park every day for 10 miles. Or if you're still in the campus, you just need the bike. So this is like you can have a variety of those your family and you can use depending on the situation. Pick the best one you want to choose. But I guess nobody just wants to have a truck, I think driving for all kinds of tasks or so. I think the same thing is going to happen in the AI world is there I don't think a one size fits all. Make sense? Because just economic point of view in the consumer market actually, you should push to the extreme efficiency. One example is when we're looking at our kitchen, okay or bakeware cookware, how many of our ideal, I just bought one package and they gave me a very small hand for cooking eggs. It's very fancy, but I feel it's probably too much to the extreme of my per task specific specific optimization. I don't use them him very much, maybe once a month. Right. So that's an analogy and I think it's it's been happening in when enterprise pride I started to think about how to leverage the innovation in our world. How much specific in the world to go down to to pick and choose the most cost efficient was of the highest quality for their product. I don't think they're gonna go that one is going to evolve into either of those extremes. Because it makes a lot of economic sense for, for us to customize, right because one big thing is going to be so complex and for all the problems we don't solve for particular product, but if you over specialize that you have imagined now, right so even manage as you go from any person that they are managing 1000s If not 10s upon hundreds of model in this, it's impossible for them to move fast if they already get into that situation. So I think I really do see that I think of Volunteer Task showing is going to is going to happen, right because it makes sense that you classify your product needs based on the towards specific specific models and ensure optimal customer uses. So that's kind of I think that that will be very interesting to see how eventually evolves and follows itself when innovation happens. shuffled this spectrum, and I think we're gonna exploit
Jianchang Mao 10:56
anything to that? No, I think like, I agree with all the three right so I think like on the cost as like, when we're talking about him and people are the size and weight need this. Bla bla bla but let's look at 94 and this will look next year there will be something else right I mean, maybe it's more complex, another site model itself as complex or the other side of it. We also agree with JC that these are multitaskers right? So they can you can iterate faster, you can go to market faster. Once you have enough adoption, you can go with a smaller model and optimize
11:43
them really sounds promising. But maybe the last question and I challenge you to work on tools, innovative exponential on your opinion. This segment investigates companies investing in terms of its finances, but as we've heard GM seeds are changing so fast, right? So after like two years investment, six months investment, you know, maybe something blew up. Your investment is right, like plenty of waste. So, how do we do make decisions about Shall we do something now, or shall we wait in this very fast changing events? So how do you make this one not two minutes
12:43
I don't have a glass
12:51
maybe it's my nature but I think this is one of those cases where you have to get in and learn how to assess like, I mean, it's at all levels. It's even the UX. If you think about when mobile the first mobile UX worked on some very primitive. We didn't know it took us several years to figure out the mobile UX. So I don't think sitting and trying to figure out what it will look like in a vacuum is going to work. So we do have to invest somewhere. flexibility is key. So for example, some of the ways that we design samskaras you can you can switch between models, but your global reach. So there are certain ways in which you can train, like your applications should be constructed in a way that it can switch between models. And it should work workflow, and maybe applications to be constructed where they could use different models for different applications use many, many models. So there's certain I mean, at the end of the day applications where rubber meets the road. That's how does that evolve is the question we need to ask. And to me a lot of flexibility there. And is a very key piece. And I think having the agility, the ability to have an agile murlocs those are the key things. And like someone was saying before the thinking of investing, being broadly interested better. I mean, I'm not talking about the textile industry broadly investor. You don't know which race is gonna win. And there's like, there's actually 10 Different games being played as long as you are. There's a tendency to think there's one race it is
14:44
make sense to see.
14:46
Yeah. A couple of trends are very clear to me. One is foundation models, enabling technologies were enabled by support applications. So the demand for the foundation of Moscow and the fine tune foundation model will increase to be second trend. The model is going to get bigger and bigger. Especially more advanced functions will get new news. So I think it should be future proof or even best to be scalable, flexible infrastructure, and ideally should be elastic, depending on the workload of the community choose.
15:44
Various possible
15:46
revenue. Certainly investment is in my opinion is made up some very advanced experimentation and post your infrastructure and these allow agility developer agility, quickly prototyping and testing you
16:15
I think like a lot of us do build products. Come back and look at the product. Just go in product and it's because like what Robbie was talking about. With this, there's so much even if it's a trial, even if there's a small group of people who are explosive. There's so much more than that. But don't break it and
16:48
so yeah, so basically, I have to separation we shall go to one is actually pretty fast to test
17:07
the loss of money and the second failure says he has no expiration and then when you deploy a full scale and and then cost of storage. So the optimization across these two admission loops are different. I think it's important to like for current, most of the company actually are more or less enforced. I wish to play now, because we have seen the possibility of those who wanted a model and what power they had. And all these things and try them out to at least one is can it improve current practice? Or can it create new hires to make the mental health offering infrastructure? So we want to actually enable this last iteration and really show you unleash possibilities, innovations at a product level without all the companies that you can choose what infrastructure is going to humans to beautiful so because I would make sense of it is to they're talking about extreme what was the other is production, production optimization. They are at odds to each other bit With tensions because the flexibility and enable innovation that means it's actually entered into a breakthroughs is able to bring current paradigm and production optimization tend to optimize for current partner so that's why they all have asked each other and that's what I like based on my experience beauty pikachu in the past five years we costume to pick the design point between flexibility and performance optimization. But the lesson learned is we should be glad all these into one solution. In which caused us to separate out into two at least two different phases of the problem the other group, and then there's so as a kind of in the future. Definitely firewalls is deep in the trenches on solving those problems under visual, many other tools. And all the cops come to you when they can many companies are thinking about that. And many other startups are thinking about that. So I think it's very exciting looking into what's going to come in the future and how the whole entire community is coming together.
19:50
matter what should just something's bad. I will think about it this way, like which problems do I want to invest in solving? Because how we solve that problem is going to change the way we think about what problems invest in, invest in plants get good approaches to solve the problem. That's a good way to think about coding. And the next level is what new problems will be created if we solve the current problem.
20:25
Yeah, totally. I think that this is really the aerodynamics and a lot of things, you know, the two sides need to come back with companies that have used it. Seeing all these different use cases and the leads and also the challenge, but due to time constraints, we probably won't have time to dive into all these questions. But our panelists, waiting sites
