AI trends revolutionizing industries through data engineering

Artificial intelligence is a game-changing technology in the enterprise world, increasingly adopted for its potential to enhance human capabilities. AI trends are driving industrial transformation across sectors.

Large language models, infrastructure optimization, generative AI and platforms for scaling have already hit the road when it comes to notable AI trends, according to Anand Pradhan (pictured), senior director of technology at the AI Center of Excellence at Intercontinental Exchange — he also runs the NYSE Launchpad Lab.

“Anytime you see people talk about the large language model and generative AI, what I notice is how these products are being built so the data that goes into the large language model,” he said. “The other aspect is all the infrastructure that is needed to be built. There are layers of companies who are working on that space, optimizing hardware, memory, chipset, storage. The last thing is anybody who has a platform to scale, those are the companies who are going to, at the end of the day, benefit a lot because the large language models will keep changing or keep creating.”

Pradhan spoke with theCUBE Research’s John Furrier at the Media Week NYC: theCUBE + NYSE Wired 2024 event, during an exclusive broadcast on theCUBE, SiliconANGLE Media’s livestreaming studio. They discussed why AI trends are a force to be reckoned with since they are opening new avenues for innovation.

How data engineering powers AI trends

Since AI is transformative thanks to the automation of repetitive tasks, enhanced personalization and improved decision-making, data engineering acts as the foundation of the algorithms behind the scenes, making it a crucial aspect in AI trends, Pradhan pointed out. 

“There are very interesting companies who are trying to streamline the data, taking many sources of data and getting streamlined pipelines,” he said. “I met a company yesterday, it just blew my mind what they’re doing in terms of data. They’re providing data to the large language model trainers or basically OpenAI or Lama, Meta and things like that. Spotify has one of the best data tech stack, which is there for research.”

As data-driven and automated decision-making remains a key focus across industries, AI decision engines are gaining steam. Pradhan attributes its rise to its predictive capabilities, scalability, adaptability and consistency.

“Coolest thing always is if somebody can build a decision engine using AI,” he said. “I have seen something where literally a decision can be made using AI. I understand what it takes to build that. We have attempted once … we did a full-blown experiment to put different AI models together to make a decision and that’s something super cool.”

Here’s the complete video interview, part of SiliconANGLE’s and theCUBE Research’s coverage of the Media Week NYC: theCUBE + NYSE Wired 2024 event: 

Photo: SiliconANGLE