亚洲国产日韩欧美一区二区三区,精品亚洲国产成人av在线,国产99视频精品免视看7,99国产精品久久久久久久成人热,欧美日韩亚洲国产综合乱

Table of Contents
Introduction to the Groq LPU Inference Engine
Install Jan AI
Create Groq Cloud API
Using Groq with Jan AI
Using Groq in VSCode
Conclusion
Home Technology peripherals AI Step-by-step guide to using Groq Llama 3 70B locally

Step-by-step guide to using Groq Llama 3 70B locally

Jun 10, 2024 am 09:16 AM
ai llm large language model

本地使用Groq Llama 3 70B的逐步指南

Translator| Bugatti

Reviser| Chonglou

This article describes how to use the Groq LPU inference engine to generate ultra-fast responses in Jan AI and VSCode.

本地使用Groq Llama 3 70B的逐步指南

Everyone is working on building better large language models (LLM), such as Groq focusing on the infrastructure side of AI. Rapid response from these large models is key to ensuring that these large models respond more quickly.

This tutorial will introduce the Groq LPU parsing engine and how to access it locally on your laptop using the API and Jan AI. This article will also integrate it into VSCode to help us generate code, refactor code, enter documentation and generate test units. This article will create our own artificial intelligence programming assistant for free.

Introduction to the Groq LPU Inference Engine

The Groq LPU (Language Processing Unit) inference engine is designed to generate fast responses for computationally intensive applications (such as LLMs) that have sequential components.

Compared with CPU and GPU, LPU has more powerful computing power, which reduces the time required to predict words and greatly speeds up the generation of text sequences. Furthermore, LPU can also handle memory bottlenecks compared to GPU, thus providing better performance on LLM.

In short, Groq LPU technology makes your LLM super fast for real-time AI applications. You may wish to read the Groq ISCA 2022 paper (https://wow.groq.com/isca-2022-paper/) to learn more about the LPU architecture.

Install Jan AI

Jan AI is a desktop application that runs open source and proprietary large language models natively. It is available for download in Linux, macOS and Windows versions. We will download Jan AI and install it in Windows. To do this, go to https://github.com/janhq/jan/releases and click on the file with the ".exe" extension.

本地使用Groq Llama 3 70B的逐步指南

If you want to use LLM locally for enhanced privacy, please read the "5 Ways to Use LLM on Your Laptop" blog post (https://www.kdnuggets. com/5-ways-to-use-llms-on-your-laptop) to start using the most advanced open source language models.

Create Groq Cloud API

To use Grog Llama 3 with Jan AI, we need an API. To do this, we will go to https://console.groq.com/ and create a Groq Cloud account.

If you want to test the various models provided by Groq, you can do this without any setup, just go to the Playground tab, select the model, and add user input.

In this example, it is very fast, generating 310 tokens per second, which is the fastest I have seen so far. Not even Azure AI or OpenAI can achieve this type of results.

本地使用Groq Llama 3 70B的逐步指南

To generate an API key, click the "API Key" button on the left panel, then click the "Create API Key" button to create and copy the API key .

本地使用Groq Llama 3 70B的逐步指南

Using Groq with Jan AI

In the next step, we will paste the Groq Cloud API key into the Jan AI application.

Launch the Jan AI application, go to settings, select the "Groq Inference Engine" option in the extension section, and add the API key.

本地使用Groq Llama 3 70B的逐步指南

Then, return to the thread window. In the Model section, select Groq Llama 370B in the "Remote" section to start typing the prompt.

本地使用Groq Llama 3 70B的逐步指南

The responses are generated so fast that I can’t keep up.

本地使用Groq Llama 3 70B的逐步指南

Note: The free version of this API has some limitations. Please visit https://console.groq.com/settings/limits to learn more about them.

Using Groq in VSCode

Next, we will try to paste the same API key into the CodeGPT VSCode extension and build our own free AI programming assistant.

Search for the CodeGPT extension in the extensions tab and install it.

本地使用Groq Llama 3 70B的逐步指南

#The CodeGPT tab will appear allowing you to select a model provider.

本地使用Groq Llama 3 70B的逐步指南

When you select Groq as the model provider, it will ask you for your API key. Just paste the same API key and we are good to go. You can even generate another API key for CodeGPT.

本地使用Groq Llama 3 70B的逐步指南

Now we will ask it to code the snake game. It only took 10 seconds to generate and run the code.

本地使用Groq Llama 3 70B的逐步指南

Below is a demonstration of our snake game.

本地使用Groq Llama 3 70B的逐步指南

You might as well learn about the top five AI programming assistants (https://www.kdnuggets.com/top-5-ai-coding-assistants-you-must-try) and become AI-driven developers and data scientists. Remember, AI is meant to help us, not replace us, so be open to it and use it to improve your coding.

Conclusion

In this tutorial we learned about the Groq inference engine and how to access it locally using the Jan AI Windows application. Finally, we integrated it into our workflow by using the CodeGPT VSCode extension, which is awesome. It generates responses in real-time for a better development experience.

Original title: Using Groq Llama 3 70B Locally: Step by Step Guide, author: Abid Ali Awan

Link: https://www.kdnuggets.com/using-groq-llama- 3-70b-locally-step-by-step-guide.

To learn more about AIGC, please visit:

51CTO AI.x Community

https://www.51cto.com/ aigc/

The above is the detailed content of Step-by-step guide to using Groq Llama 3 70B locally. For more information, please follow other related articles on the PHP Chinese website!

Statement of this Website
The content of this article is voluntarily contributed by netizens, and the copyright belongs to the original author. This site does not assume corresponding legal responsibility. If you find any content suspected of plagiarism or infringement, please contact admin@php.cn

Hot AI Tools

Undress AI Tool

Undress AI Tool

Undress images for free

Undresser.AI Undress

Undresser.AI Undress

AI-powered app for creating realistic nude photos

AI Clothes Remover

AI Clothes Remover

Online AI tool for removing clothes from photos.

Clothoff.io

Clothoff.io

AI clothes remover

Video Face Swap

Video Face Swap

Swap faces in any video effortlessly with our completely free AI face swap tool!

Hot Tools

Notepad++7.3.1

Notepad++7.3.1

Easy-to-use and free code editor

SublimeText3 Chinese version

SublimeText3 Chinese version

Chinese version, very easy to use

Zend Studio 13.0.1

Zend Studio 13.0.1

Powerful PHP integrated development environment

Dreamweaver CS6

Dreamweaver CS6

Visual web development tools

SublimeText3 Mac version

SublimeText3 Mac version

God-level code editing software (SublimeText3)

Hot Topics

PHP Tutorial
1488
72
How to use PHP to build social sharing functions PHP sharing interface integration practice How to use PHP to build social sharing functions PHP sharing interface integration practice Jul 25, 2025 pm 08:51 PM

The core method of building social sharing functions in PHP is to dynamically generate sharing links that meet the requirements of each platform. 1. First get the current page or specified URL and article information; 2. Use urlencode to encode the parameters; 3. Splice and generate sharing links according to the protocols of each platform; 4. Display links on the front end for users to click and share; 5. Dynamically generate OG tags on the page to optimize sharing content display; 6. Be sure to escape user input to prevent XSS attacks. This method does not require complex authentication, has low maintenance costs, and is suitable for most content sharing needs.

PHP calls AI intelligent voice assistant PHP voice interaction system construction PHP calls AI intelligent voice assistant PHP voice interaction system construction Jul 25, 2025 pm 08:45 PM

User voice input is captured and sent to the PHP backend through the MediaRecorder API of the front-end JavaScript; 2. PHP saves the audio as a temporary file and calls STTAPI (such as Google or Baidu voice recognition) to convert it into text; 3. PHP sends the text to an AI service (such as OpenAIGPT) to obtain intelligent reply; 4. PHP then calls TTSAPI (such as Baidu or Google voice synthesis) to convert the reply to a voice file; 5. PHP streams the voice file back to the front-end to play, completing interaction. The entire process is dominated by PHP to ensure seamless connection between all links.

How to use PHP combined with AI to achieve text error correction PHP syntax detection and optimization How to use PHP combined with AI to achieve text error correction PHP syntax detection and optimization Jul 25, 2025 pm 08:57 PM

To realize text error correction and syntax optimization with AI, you need to follow the following steps: 1. Select a suitable AI model or API, such as Baidu, Tencent API or open source NLP library; 2. Call the API through PHP's curl or Guzzle and process the return results; 3. Display error correction information in the application and allow users to choose whether to adopt it; 4. Use php-l and PHP_CodeSniffer for syntax detection and code optimization; 5. Continuously collect feedback and update the model or rules to improve the effect. When choosing AIAPI, focus on evaluating accuracy, response speed, price and support for PHP. Code optimization should follow PSR specifications, use cache reasonably, avoid circular queries, review code regularly, and use X

PHP realizes commodity inventory management and monetization PHP inventory synchronization and alarm mechanism PHP realizes commodity inventory management and monetization PHP inventory synchronization and alarm mechanism Jul 25, 2025 pm 08:30 PM

PHP ensures inventory deduction atomicity through database transactions and FORUPDATE row locks to prevent high concurrent overselling; 2. Multi-platform inventory consistency depends on centralized management and event-driven synchronization, combining API/Webhook notifications and message queues to ensure reliable data transmission; 3. The alarm mechanism should set low inventory, zero/negative inventory, unsalable sales, replenishment cycles and abnormal fluctuations strategies in different scenarios, and select DingTalk, SMS or Email Responsible Persons according to the urgency, and the alarm information must be complete and clear to achieve business adaptation and rapid response.

How to use PHP to combine AI to generate image. PHP automatically generates art works How to use PHP to combine AI to generate image. PHP automatically generates art works Jul 25, 2025 pm 07:21 PM

PHP does not directly perform AI image processing, but integrates through APIs, because it is good at web development rather than computing-intensive tasks. API integration can achieve professional division of labor, reduce costs, and improve efficiency; 2. Integrating key technologies include using Guzzle or cURL to send HTTP requests, JSON data encoding and decoding, API key security authentication, asynchronous queue processing time-consuming tasks, robust error handling and retry mechanism, image storage and display; 3. Common challenges include API cost out of control, uncontrollable generation results, poor user experience, security risks and difficult data management. The response strategies are setting user quotas and caches, providing propt guidance and multi-picture selection, asynchronous notifications and progress prompts, key environment variable storage and content audit, and cloud storage.

PHP integrated AI speech recognition and translator PHP meeting record automatic generation solution PHP integrated AI speech recognition and translator PHP meeting record automatic generation solution Jul 25, 2025 pm 07:06 PM

Select the appropriate AI voice recognition service and integrate PHPSDK; 2. Use PHP to call ffmpeg to convert recordings into API-required formats (such as wav); 3. Upload files to cloud storage and call API asynchronous recognition; 4. Analyze JSON results and organize text using NLP technology; 5. Generate Word or Markdown documents to complete the automation of meeting records. The entire process needs to ensure data encryption, access control and compliance to ensure privacy and security.

How to build an online customer service robot with PHP. PHP intelligent customer service implementation technology How to build an online customer service robot with PHP. PHP intelligent customer service implementation technology Jul 25, 2025 pm 06:57 PM

PHP plays the role of connector and brain center in intelligent customer service, responsible for connecting front-end input, database storage and external AI services; 2. When implementing it, it is necessary to build a multi-layer architecture: the front-end receives user messages, the PHP back-end preprocesses and routes requests, first matches the local knowledge base, and misses, call external AI services such as OpenAI or Dialogflow to obtain intelligent reply; 3. Session management is written to MySQL and other databases by PHP to ensure context continuity; 4. Integrated AI services need to use Guzzle to send HTTP requests, safely store APIKeys, and do a good job of error handling and response analysis; 5. Database design must include sessions, messages, knowledge bases, and user tables, reasonably build indexes, ensure security and performance, and support robot memory

How to use PHP to call AI writing auxiliary tools PHP improves content output efficiency How to use PHP to call AI writing auxiliary tools PHP improves content output efficiency Jul 25, 2025 pm 08:18 PM

When choosing an AI writing API, you need to examine stability, price, function matching and whether there is a free trial; 2. PHP uses Guzzle to send POST requests and uses json_decode to process the returned JSON data, pay attention to capturing exceptions and error codes; 3. Integrating AI content into the project requires an audit mechanism and supporting personalized customization; 4. Cache, asynchronous queue and current limiting technology can be used to optimize performance to avoid bottlenecks due to high concurrency.

See all articles