This is "The AI Economy," a weekly LinkedIn-first newsletter about AI's influence on business, work, society and tech and written by Ken Yeung. Sign up here.
As I’m writing this week’s newsletter and listening to Taylor Swift’s latest album—which has its own AI controversy—I’m astonished at how jam-packed the news cycle has been. Stanford University released a massive and informative report that gives us a pulse on where AI stands today. Google is undergoing a reorganization to align its AI efforts better. And AI is all the talk at this year’s TED conference.
But Meta is what everyone’s talking about. The company released its next-generation large language model, Llama 3, dominating the conversation before OpenAI announces GPT-5 and Microsoft and Google hold their annual developer conferences.
Let’s take a look at what makes Llama 3 special. And stick around for nearly 50 AI headlines you may have missed!
The Prompt
Part of Meta’s growing line of AI tools, Llama 3 comes in two sizes: Llama 3 8B, which features eight billion parameters and the more powerful Llama 3 70B, which has 70 billion parameters. Both versions are available today or soon will be on Amazon Web Services, Databricks, Google Cloud, Hugging Face, Kaggle, IBM Watson X, Microsoft Azure, Nvidia NIM, and Snowflake.
Like its predecessors, Llama 3 is open-sourced. “We believe these are the best open-source models of their class, period,” Meta claims. “In support of our longstanding open approach, we’re putting Llama 3 in the hands of the community. We want to kickstart the next wave of innovation in AI across the stack—from applications to developer tools to [evaluations] to inference optimizations and more.”
Experience Llama 3 on Meta AI
While the models are available for downloading, Llama 3 is already in use, powering the Meta AI experience on Facebook, Instagram, WhatsApp, Messenger and the web. Eventually, the company says it will test multimodal Meta AI on its Ray-Ban smart glasses.
Meta AI is the company’s chatbot—the thing you see embedded into the search bar of all Meta apps. It was introduced last September and is positioned as a competitor to ChatGPT. Chief Executive Mark Zuckerberg tells The Verge that Meta AI aims to be “the most intelligent AI assistant that people can freely use across the world. With Llama 3, we basically feel like we’re there.”
It’s now available outside of the U.S., with English rollouts to Australia, Canada, Ghana, Jamaica, Malawi, New Zealand, Nigeria, Pakistan, Singapore, South Africa, Uganda, Zambia, and Zimbabwe.
Llama 3 Architecture
The new model uses a tokenizer with a vocabulary of 128,000 tokens to encode language more efficiently. It also improves inference efficiency thanks to grouped query attention (GQA).
Meta pre-trained Llama 3 on over 15 trillion tokens, which it says were collected from publicly available sources. That dataset is seven times larger than the one used for Llama 2 and includes four times more code. Over five percent of the dataset consists of non-English data covering over 30 languages, a step the company made to prepare its model for multilingual use cases. However, it warned Llama 3’s performance in non-English languages to be on par with English.
The company says training Llama 3’s largest models required a combination of three types of parallelization (a form of computing), synchronizing data with model and pipeline. These training runs were done on two custom-built 24K GPU clusters.
Benchmarking
Meta claims its new LLMs offer performance comparable to or better than that of Google Gemini, Anthropic’s Claude 3 Sonnet, and Mistral’s 7B Instruct. As VentureBeat notes, Llama 3 “does well at multiple-choice questions (MMLU) and coding (HumanEval), but the 70B is not as strong as Gemini Pro 1.5 at solving math word problems (MATH), nor at graduate-student-level multiple-choice questions (GPQA).”
It also notes that the Llama 3 8B outperforms Gemma 7B and Mistral 7B across many benchmarks, including grade school math questions.
Take a look at the following charts to see more about how Llama 3 compares to other leading AI models:
Questions surrounding data sources
What specific sources did Meta use to train Llama 3? The company hasn’t disclosed the sites, only to say it pulled from “publicly available sources.” This could mean, among other things, the content you share across the Meta family of apps, from Facebook and Instagram to WhatsApp and Messenger. So, double-check those privacy settings to ensure you’re sharing what you want to share.
Not stopping with Llama 3
“The Llama 3 8B and 70B models mark the beginning of what we plan to release for Llama 3,” Meta reveals. It plans to debut more models that are capable of multimodality, able to converse in multiple languages, have a longer context window and have stronger overall capabilities.
If you think 70 billion parameters are big, wait until Meta releases its 400 billion parameter model, which is still being trained.
Further Reading:
- Podcast: How Meta built Llama 3 (Big Tech War Stories/Alex Kantrowitz)
- Meta is already working on a more powerful successor to Llama 3 (Wired)
- Meta says Llama 3 beats most other models, including Gemini (The Verge)
- Meta brings real-time AI image generation to WhatsApp (The Verge)
- What did Elon Musk have to say about Llama 3? (VentureBeat)
Today’s Visual Snapshot
The Stanford Institute for Human-Centered Artificial Intelligence released its 2024 AI Index Report this week. For seven years, it has explored the technology’s influence on society. This year, it broadened its research to probe trends such as technical advancements in AI, public perceptions, and the geopolitical dynamics surrounding its development.
It’s impossible to summarize everything in the study, which spans over 500 pages. However, IEEE Spectrum has replicated 15 featured charts summarizing the current state of AI. Below are several of note:
Google is the market leader in creating foundational models, surpassing Meta, Microsoft and OpenAI. These models are used as the backbone for AI apps, such as OpenAI’s GPT-4, which powers ChatGPT. As IEEE Spectrum notes, many of these foundational models are owned by “industry,” or commercial entities. A few were created by academic institutions such as Stanford University and the University of California, Berkeley.
Building and training AI models are expensive, and it shouldn’t surprise anyone that only a few companies can develop them. But how much is being spent? Google invested over $191 million into its most powerful LLM, Gemini Ultra. By comparison, OpenAI is believed to have invested over $78 million into GPT-4. And the 2017 transformer model created by Google that helped kickstart the LLM evolution? $930 went into training it.
The last chart I’m highlighting compares open and closed models. Which foundation model type performs the best? IEEE Spectrum notes that the AI Index looks at the trend of released open and closed models. The above chart suggests that closed models outperform open ones across multiple standard benchmarks. However, the report doesn’t address the core debate around these model types: Which is better for security and innovation?
Quote This
“We thought it was going to be something that had to do with training large models. At the time I thought it was probably going to be something that had to do with content. It’s just the pattern matching of running the company, there’s always another thing. At that time I was so deep into trying to get the recommendations working for Reels and other content. That’s just such a big unlock for Instagram and Facebook now, being able to show people content that’s interesting to them from people that they’re not even following.
But that ended up being a very good decision in retrospect. And it came from being behind. It wasn’t like “oh, I was so far ahead.” Actually, most of the times where we make some decision that ends up seeming good is because we messed something up before and just didn’t want to repeat the mistake.”
— Mark Zuckerberg explaining that Meta purchased 350,000 Nvidia H100 GPUs this year to power Reels, but then found it more useful for its AI efforts (Dwarkesh Patel Podcast)
This Week’s AI News
🏭 Industry Insights
- Stanford University research shows “startlingly rapid” progress in AI, with systems nearly matching or exceeding human performance in reading comprehension, image classification and competition-level math (Scientific American)
- A look at the new AI tools from Google’s Jigsaw subsidiary that could help unite a fractious internet (TIME)
- The rise of the chief AI officer (Financial Times)
- Investor Peter Thiel claims AI will be “worse” for math professionals than writers (Business Insider)
- The European Union will not launch a probe into Microsoft’s $13 billion investment in OpenAI (Bloomberg)
- Generative AI’s open secret: Everyone is copying everyone else (The Information)
🤖 Machine Learning
- The Allen Institute for AI (AI2) updates its OLMo model with a more diversified dataset and two-stage training curriculum (VentureBeat)
- Nonprofit MLCommons announces new benchmark to measure the safety of AI systems, evaluating a large language model’s response to prompts across various “hazard categories” (Silicon Angle)
- Meta and USC researchers propose the Megalodon machine learning model to challenge the Transformer (VentureBeat)
- Zyphra releases an SSM-hybrid foundation model called Zamba to bring AI to more devices (VentureBeat)
✏️ Generative AI
- Microsoft researchers introduce VASA-1, an AI model that turns a single portrait photo and an audio file into a realistic talking face video (Tom’s Guide)
- Adobe says it’s exploring opening up its video tools to third-party generative AI tools such as OpenAI’s Sora (Reuters)
- Hugging Face releases benchmark to test generative AI on health tasks (TechCrunch)
- Brave launches real-time privacy-focused AI answer engine built from scratch and doesn’t use any Big Tech search technology (VentureBeat)
- Who’s ready for a chatbot version of their favorite Instagram influencer? (The New York Times)
☁️ Enterprise
- Intel, Cloudera, the Linux Foundation and other organizations commit to building open generative AI tools for the enterprise (TechCrunch)
- Zendesk unveils an AI-powered CX platform with sophisticated agents and intelligent copilots (VentureBeat)
⚙️ Hardware and Robotics
- Boston Dynamics unveils “stronger” fully electric humanoid robot called Atlas (CBS News)
- A look at how large language models are ushering in a new era of robotics (VentureBeat)
- AI was supposed to make police bodycams better, but what happened? (MIT Technology Review)
- Limitless debuts a wearable pendant that records everything you hear and then uses AI to help you understand it (The Verge)
- U.S. Air Force reveals AI-controlled fighter jet successfully faced a human pilot during an in-air dogfight (The Verge)
🔬 Science and Breakthroughs
- Alphabet’s X moonshot factory unveils Project Bellwether which uses AI to help predict natural disasters (TechCrunch)
- Generative AI is primarily trained in English, and that’s negatively impacting thousands of languages (The Atlantic)
- Dartmouth researchers look to meld therapy apps with modern AI (NBC News)
- Intel unveils Hala Point, a large neuromorphic computing system designed to aid research into future brain-inspired AI (VentureBeat)
💼 Business and Marketing
- Google restructures to consolidate teams building AI models across its Research and DeepMind divisions in a bid to better develop AI products (Reuters)
- Stability AI lays off 10 percent of staff after the departure of controversial CEO Emad Mostaque (CNBC)
- AI presentation tool maker Tome restructures, laying off 20 percent of employees as it sets sights on sales teams (Semafor)
- Will AI-generated models bring more or less diversity to the fashion industry? (Associated Press)
- Secretive and controversial software company Palantir is reportedly pitching ad agencies on its AI technology (Marketing Brew)
- LinkedIn’s AI-powered Collaborative Articles feature: ‘Cesspool of AI crap’ or slam-dunk success (Fortune)
📺 Media and Entertainment
- AI made these movies shaper. Critics say it ruined them. (The New York Times)
- Netflix’s documentary “What Jennifer Did” uses AI images to create a false historical record (404 Media)
- A24 is under fire after using AI-generated ads to promote “Civil War” (Futurism)
💰 Funding
- AI industry is facing a reckoning as global investment in the sector fell for the second year in a row (TechCrunch)
- Mistral is reportedly seeking to raise capital at a $5 billion valuation (The Information) — supposedly up to $2 billion is being sought (Business Insider)
- DeepMind CEO Demis Hassabis says Google will outpace Microsoft in AI investing (The Next Web)
- NeuBird raises $22 million for its generative AI solution for complex cloud-native environments (TechCrunch)
⚖️ Copyright and Regulatory Issues
- When Elisa Shupe used ChatGPT to write a novel, it was used to make the U.S. Copyright Office overturn its policy on work made with AI — she won, but there’s a catch (Wired)
- Snap plans to add watermarks to AI-generated images on its platform (TechCrunch)
- Glaze Project releases a new version of its anti-AI scraping tool and plans to soon add video protections (VentureBeat)
- Former OpenAI director Helen Toner calls for AI leaders to disclose details of their most advanced systems and make them available to outside auditing (Axios)
💥 Disruption and Misinformation
- Google hosted over 100 YouTube videos promoting AI deepfake porn but has since removed them (Forbes)
- Creating sexually explicit deepfake images to be made a criminal offense in the UK (The Guardian)
- Beware these real-time deepfake romance scams (Wired)
🎧 Podcasts
- GeekWire Podcast: Amazon CTO Werner Vogels on AI’s rapid progress and impact on society (GeekWire)
- Decoder: Dropbox CEO Drew Houston on embracing AI and remote work (The Verge)
End Output
Thanks for reading. Be sure to subscribe so you don’t miss any future issues of this newsletter.
Did you miss any AI articles this week? Fret not; I’m curating the big stories in my Flipboard Magazine, “The AI Economy.”
Connect with me on LinkedIn and check out my blog to read more insights and thoughts on business and technology.
Do you have a story you think would be a great fit for “The AI Economy”? Awesome! Shoot me a message – I’m all ears!
Until next time, stay curious!
Subscribe to “The AI Economy”
New issues published on Fridays, exclusively on LinkedIn
Leave a Reply
You must be logged in to post a comment.