Celery is a practical tool for asynchronous tasks in Python, which can improve response speed and system throughput. 1. Install Celery and select Redis or RabbitMQ as broker; 2. Define the task module and call it asynchronously through .delay(); 3. Use result backend (such as Redis) to query the task results; 4. Configure the beat module to implement timing tasks and run them in conjunction with worker. It is suitable for handling time-consuming operations, timing tasks, email sending, image processing and other scenarios. The key is to understand the collaboration mechanism of broker, worker and backend.

Celery is a very practical tool for doing asynchronous tasks in Python. It can help you strip time-consuming operations from the main process and improve response speed and system throughput. If your project needs to handle timing tasks, sending emails, image processing or data calculation, Celery is suitable.

Installation and basic configuration

The first step to using Celery is to install it, and it also requires a message broker, most commonly used is Redis or RabbitMQ.

 pip install celery

Then you need a task module, such as tasks.py :

 from celery import Celery

app = Celery('tasks', broker='redis://localhost:6379/0')

@app.task
def add(x, y):
    return xy

Start worker to execute tasks:

 celery -A tasks worker --loglevel=info

This command will start a worker to listen to the task queue, and will be executed once a task comes in.

Methods for calling tasks asynchronously

After defining the task, you do not need to call the function directly, but trigger it asynchronously through the .delay() method.

For example in Flask:

 from tasks import add

@app.route('/add')
def do_add():
    result = add.delay(4, 5)
    return f"Task ID: {result.id}"

In this way, the request will be returned immediately, and the addition operation will be executed asynchronously in the background. You can query the status of the result by task ID.

• Use .apply_async() to control parameters and scheduling more flexibly
• You can set advanced options such as retry mechanism, timeout time, priority, etc.
• If you want to perform tests synchronously, you can use .apply() (not recommended for production)

Use Result Backend to get task results

By default, the task is finished after execution, and you cannot know its status or result. At this time, you need to set result backend.

Celery supports a variety of backends, such as Redis, database, RPC, etc. Take Redis as an example, when initializing:

 app.conf.update(
    result_backend='redis://localhost:6379/0'
)

After that, you can query the task results anywhere:

 from celery.result import AsyncResult

result = AsyncResult(task_id)
print(result.state) # View status print(result.result) # View results

• Some backends may have limited performance, such as the database is not suitable for high concurrency scenarios.
• If you only care about whether the task is completed, you don't have to turn on result backend
• Redis or dedicated storage solutions are recommended for high concurrency.

How to match scheduled tasks

In addition to asynchronous tasks, Celery also supports periodic tasks (Periodic Tasks). The celery beat module is required.

Define the task and scheduler first:

 from celery.schedules import crontab

app.conf.beat_schedule = {
    'add-every-30-seconds': {
        'task': 'tasks.add',
        'schedule': 30.0,
        'args': (16, 16)
    },
}

Then start beat:

 celery -A tasks beat

It can also be run with worker:

 celery -A tasks worker --beat

• Timed task information can be persisted into the database (using django-celery-beat )
• The scheduling frequency can be used in crontab format, such as when every day
• Be careful to ensure that beat and worker run at the same time, otherwise the timing tasks will not be processed.

Basically that's it. Celery is powerful but has a slightly complex configuration. The key is to understand the relationship between broker, worker, and backend. You may encounter some problems at the beginning, such as not executing the task or not getting the results. You can locate the cause by checking the log more. After using it well, asynchronous task management will become very easy.

The above is the detailed content of Python Celery for Asynchronous Tasks. For more information, please follow other related articles on the PHP Chinese website!