asyncio for bots and scrapers

asyncio is how Python handles lots of slow, waiting-on-the-network work without threads: it starts many operations, and while each waits for a reply, others run. It's the engine under discord.py (every command handler is async) and the fastest way to write a scraper that fetches hundreds of pages. This guide covers the async patterns that fit those Falix workloads, and the single mistake that stalls the whole thing.

async def, await, and the loop

An async def function is a coroutine — calling it doesn't run it, it hands back something you await. await pauses the coroutine until the result is ready, freeing the single thread to run other coroutines meanwhile. An event loop drives them all; asyncio.run() starts one:

import asyncio

async def fetch(name):
    await asyncio.sleep(1)      # stands in for a network call
    return f"done: {name}"

async def main():
    result = await fetch("one")
    print(result, flush=True)

asyncio.run(main())

🎯 Good to know: On Falix, print with flush=True (or print(..., flush=True)) for long-running async apps, so your logs reach the Console immediately instead of sitting in Python's buffer.

Bot-shaped: everything is already async

In discord.py you don't call asyncio.run yourself — the library runs the loop. Your job is to keep handlers async and await the bot's calls:

@bot.tree.command(name="ping", description="Check the bot")
async def ping(interaction: discord.Interaction):
    await interaction.response.send_message("Pong!")

The rule that keeps a bot healthy: never block the loop. A blocking call inside a handler freezes the entire bot — it stops answering commands and can drop its Discord connection. See the trap below.

Scraper-shaped: many requests at once

Doing requests one after another wastes almost all your time waiting. asyncio.gather runs them concurrently and collects the results:

import asyncio, aiohttp

async def get(session, url):
    async with session.get(url) as resp:
        return await resp.text()

async def main():
    urls = ["https://example.com", "https://example.org", "https://example.net"]
    async with aiohttp.ClientSession() as session:
        pages = await asyncio.gather(*(get(session, u) for u in urls))
    print(f"fetched {len(pages)} pages", flush=True)

asyncio.run(main())

Ten sequential 1-second requests take ten seconds; ten with gather take about one. (aiohttp or httpx are the async HTTP clients — add one on the Packages page.)

💡 Tip: Don't fire thousands of requests at once — you'll hammer the target and hit rate limits. Cap concurrency with an asyncio.Semaphore(20) and await it inside each task.

The trap: one blocking call freezes everything

Because it's all one thread, a synchronous call that blocks stops every other coroutine dead. The three usual offenders:

If your bot periodically goes unresponsive with no error in the console, a blocking call in an async handler is the first suspect. asyncio.to_thread(...) pushes a blocking or CPU-heavy function onto a worker thread so the loop keeps turning.

Managing tasks

asyncio.create_task(...) starts a coroutine running in the background without waiting for it. Two honest cautions:

• Keep a reference to the task. A task with no reference can be garbage-collected and silently cancelled mid-run.
• Handle its exceptions. An exception in a background task that nobody awaits is easy to miss — wrap the work in try/except and log failures, or the task dies quietly.

task = asyncio.create_task(background_job())
# ... keep `task` alive; later:
await task   # surfaces any exception it raised

discord.ext.tasks wraps this pattern for bots (looping jobs like periodic announcements) and handles the reference-keeping for you.

Troubleshooting

• Bot goes silent for seconds at a time — a blocking call on the loop. Swap time.sleep/requests for their async equivalents, or offload with asyncio.to_thread.
• RuntimeWarning: coroutine '...' was never awaited — you called an async function without await. Add the await, or create_task it if it should run in the background.
• A background task's error never showed up — you didn't await it or catch its exception. Wrap it and log.
• asyncio.run() cannot be called from a running event loop — you're already inside one (e.g. discord.py runs it). Don't call asyncio.run there; just await your coroutine.

The asyncio model has a lot more depth (queues, cancellation, timeouts) — the official Python asyncio docs are the reference.

Next steps

• Host a discord.py bot
• Python on Falix
• Pinning Python requirements