The Ultimate Guide to Sending POST Requests with cURL

ByPaul Christiano Last Update on

As a web developer or data engineer, sending HTTP requests is a core part of interacting with web services and APIs. While there are many tools and libraries available for making HTTP requests, cURL (client URL) remains one of the most popular and powerful options – especially for sending POST requests.

In this comprehensive guide, we‘ll dive deep into how to use cURL to send POST requests for any use case. I‘ll share practical examples, performance tips, and best practices from my years of experience as a professional web scraper and API integrator.

By the end of this article, you‘ll be equipped with everything you need to know to become a cURL power user. Let‘s get started!

Anatomy of a cURL POST Request

At its core, an HTTP POST request consists of a few key components:

  • URL – The web address of the API endpoint or web server you want to send data to
  • Headers – Additional metadata about the request like the content type and authorization
  • Request Body – The actual payload of data you want to transmit, often in JSON or XML format
  • Options – Flags that customize cURL‘s behavior, like specifying an HTTP proxy or disabling SSL verification

Here‘s a basic example of what a complete cURL POST request looks like:


curl -X POST -H "Content-Type: application/json" -H "Authorization: Bearer $TOKEN" -d ‘{"username":"john","password":"secret"}‘ https://api.example.com/login

Breaking this down:

  • -X POST sets the HTTP method to POST
  • -H defines the Content-Type and Authorization headers
  • -d specifies the JSON payload to include in the request body
  • The final argument is the destination URL to send the request to

Whether you‘re logging into a web service, sending a payment, or creating a new record in a database, most POST requests will follow a similar structure. The headers, request body, and URL will change depending on the API, but the core syntax remains the same.

Sending Different Content Types

One of the most important headers in a POST request is Content-Type, which tells the web server how to interpret the data in the request body. Let‘s look at a few common content types and how to send them with cURL.

JSON

JSON (JavaScript Object Notation) is the most widely used format for sending data to web APIs. It represents data as key-value pairs, similar to a dictionary or hashmap in programming languages.

To post a JSON payload, we need to set the Content-Type header to application/json and provide the data as a string argument to the -d flag:


curl -X POST -H "Content-Type: application/json" -d ‘{"name": "John", "age": 30}‘ https://api.example.com/users

For readability, it‘s often better to store the JSON payload in a separate file and reference it with -d:


curl -X POST -H "Content-Type: application/json" -d @payload.json https://api.example.com/users

Where payload.json contains the JSON data:

{
"name": "John",
"age": 30
}

Form URL-Encoded Data

Form URL-encoded data is another common content type that represents data as key-value pairs concatenated with =, separated by &. It‘s often used when submitting HTML forms.

To send form-encoded data, set the Content-Type to application/x-www-form-urlencoded and pass the key-value pairs to -d:


curl -X POST -H "Content-Type: application/x-www-form-urlencoded" -d "username=john&password=secret" https://api.example.com/login

Multipart Form Data

Multipart form data allows sending a mixture of plain text values and binary files (like images or PDFs) in the same request body. Set the Content-Type to multipart/form-data and use -F to specify each part:


curl -X POST -F "[email protected]" -F "username=john" https://api.example.com/profile

This will send the contents of profile.jpg as an image file along with the username field.

Advanced Authentication

Many web APIs require authentication to access protected resources or perform actions on behalf of a user. While HTTP Basic Auth (username/password) is the simplest method, most modern APIs use more secure schemes like API keys, OAuth tokens, or client certificates. Here‘s how to authenticate with cURL for a few advanced scenarios:

Bearer Tokens

Bearer tokens, such as JWT (JSON Web Tokens), are a common way for APIs to authenticate requests. The token is included in the Authorization header with the Bearer prefix:


curl -X POST -H "Authorization: Bearer eyJhb..." https://api.example.com/user

API Keys

API keys are another simple authentication method. The key is usually sent as a custom header or query parameter:


curl -X POST -H "X-API-Key: abc123" https://api.example.com/data

OAuth

OAuth is a protocol for allowing third-party apps to access resources on behalf of a user without exposing their credentials. It involves obtaining an access token from the API provider and including it in requests.

Here‘s an example of using cURL to exchange an authorization code for an access token:


curl -X POST -d "grant_type=authorization_code&code=abc123&redirect_uri=https://myapp.com/callback" https://api.example.com/oauth/token

And then using that access token in a POST request:


curl -X POST -H "Authorization: Bearer xyz789" https://api.example.com/user/contacts

Handling Proxies and SSL

Depending on your networking environment, you may need to configure cURL to use an HTTP/HTTPS proxy or customize its SSL/TLS settings. Here are a few common options:

To send requests through a proxy server, use the -x flag:


curl -X POST -x http://proxy.company.com:3128 https://api.example.com/endpoint

To disable SSL certificate verification (not recommended for production!), use -k or –insecure:


curl -X POST -k https://self-signed.badssl.com/

You can also specify a custom certificate authority (CA) bundle for SSL verification with –cacert:


curl -X POST --cacert company-ca.pem https://api.example.com/endpoint

Performance Tips

When sending a large number of POST requests, performance can become a bottleneck. Here are a few tips to speed things up:

  • Use -C to enable compression and reduce the amount of data transferred over the network.
  • Disable progress meters and other unnecessary output with -s to reduce overhead.
  • Use persistent connections with the –keepalive-time option to reuse TCP connections
  • Parallelize requests with a tool like xargs or by using curl‘s built-in –parallel option

For example, here‘s how to send 100 compressed, parallel POST requests with progress disabled:


cat payload.json | jq -r ‘.[]‘ | curl -X POST -s -C - --parallel --parallel-max 100 https://api.example.com/submit

This will read a JSON array from the payload.json file, extract each object using jq, and submit them as the request body to the /submit endpoint using up to 100 parallel cURL processes.

According to data from Akamai, enabling HTTP/2 and compression can reduce the download time of a 1MB payload by over 60%. Multiplied across thousands or millions of API requests, this can lead to significant performance gains.

Real-World Examples

Equipped with the concepts we‘ve covered so far, let‘s walk through a few specific, real-world examples of using cURL to send POST requests.

Submitting a Contact Form

Imagine you have a contact form on your website that sends data to a CRM or customer support system. Here‘s how to replicate that form submission with cURL:


curl -X POST -d "[email protected]&message=Hello" https://mycompany.com/contact

Creating a New User Account

To programmatically create a new user account in a web application, we can send a POST request with the user‘s details:


curl -X POST -H "Content-Type: application/json" -d ‘{"username":"john","email":"[email protected]","password":"abc123"}‘ https://webapp.com/register

Uploading a File

Many APIs allow uploading files as part of a POST request. You can do this with cURL using the -F flag:


curl -X POST -F "[email protected]" -F "options={\"hasHeader\":true}" https://dataprocessor.com/uploads

Bulk API Operations

Some APIs support bulk operations, such as inserting or updating multiple records in a single request. You can use cURL to POST a JSON array of objects:


curl -X POST -H "Content-Type: application/json" -d ‘[{"id":1,"name":"John"},{"id":2,"name":"Jane"}]‘ https://api.company.com/contacts/bulk_update

These are just a few examples – the possibilities for using cURL to automate POST requests are endless!

Conclusion

In this guide, we took a comprehensive look at how to use cURL to send POST requests for almost any imaginable use case. We covered key concepts like request anatomy, content types, authentication, proxies, and optimizing for performance. I also shared practical tips and best practices based on my extensive experience scraping websites and integrating with APIs.

While tools like Postman or Insomnia offer GUI interfaces for composing HTTP requests, cURL remains an essential, universally applicable tool for any web developer or automation engineer. Its simplicity, power, and ability to run headlessly make it the ideal choice for one-off API requests or complex scripting tasks alike.

In fact, according to the GitHub State of the Octoverse report, cURL was the third most popular open source project in the world by contributor count! It shows just how indispensable this tool is to millions of developers and engineers around the globe.

So whether you‘re just starting out your web development journey or you‘re a seasoned API integrator, I hope this guide has equipped you with the skills and knowledge you need to become a cURL power user and accomplish any POST request task with ease. Now go forth and automate those APIs!

I am Paul Christiano, a fervent explorer at the intersection of artificial intelligence, machine learning, and their broader implications for society. Renowned as a leading figure in AI safety research, my passion lies in ensuring that the exponential powers of AI are harnessed for the greater good. Throughout my career, I've grappled with the challenges of aligning machine learning systems with human ethics and values. My work is driven by a belief that as AI becomes an even more integral part of our world, it's imperative to build systems that are transparent, trustworthy, and beneficial. I'm honored to be a part of the global effort to guide AI towards a future that prioritizes safety and the betterment of humanity.

Similar Posts