curl

curl - Transfers data from or to a server, using one of the protocols: HTTP, HTTPS, FTP, FTPS, SCP, SFTP, TFTP, DICT, TELNET, LDAP or FILE. (To transfer multiple files use wget or FTP.)

Although curl is widely used, a user friendly curl utility tool called httpie is open sourced and is recommanded to use.

Syntax

curl [options] [URL…]

Comon Options

-X, –request Specifies a custom request method to use when communicating with the HTTP server. The specified request method will be used instead of the method otherwise used (which defaults to GET).

-i, –include Includes HTTP response Header information in the output

–compressed Request a compressed response using one of the algorithms curl supports (gzip), and save the uncompressed document. If this option is used and the server sends an unsupported encoding, curl will report an error.(HTTP)

-d @file -d "string" --data "string"

Send the specified data in an (HTTP) POST request, in the same way that a web browser does. This will pass the data using the content-type application/x-www-form-urlencoded. Compare to -F, –form. Note that –data implies POST so –request flag can be omitted.

-d, –data is the same as –data-ascii. To post data in pure binary, use –data-binary. To URL-encode the value of a form field you may use –data-urlencode. Multiple date options will be merged together. Thus, using ‘-d name=daniel -d skill=lousy’ would generate a post that looks like ‘name=daniel&skill=lousy’. If the data starts with @, the rest should be a file name to read the data from, or - if you want curl to read the data from stdin. Multiple files can also be specified. Posting data from a file named ‘foobar’ would thus be done with –data @foobar. When –data is told to read from a file like that, carriage returns and newlines will be stripped out. If you don’t want the @ character to have a special interpretation use –data-raw instead.

-F name=@file -F name=content --form name=content

Emulate a filled-in form in which a user has pressed the submit button. This will POST data using the Content-Type multipart/form-data according to RFC 2388. This enables uploading of binary files etc. If the data starts with @, the rest should be a filename. To just get the content part from a file, prefix the file name with the symbol <. The difference between @ and < is that @ makes a file get attached in the post as a file upload, while the < makes a text field and gets the contents for that text field from a file.

-o file, –output file Write output to file instead of stdout.

-O, –remote-name

Write output to a local file named like the remote file we get. (Only the file part of the remote file is used, the path is cut off.) The remote file name to use for saving is extracted from the given URL, nothing else. Consequentially, the file will be saved in the current working directory.

-s, –silent Silent or quiet mode. Don’t show progress meter or error messages.

-I, –head Fetch the HTTP-header only! (HTTP/FTP/FILE) HTTP-servers feature the command HEAD which this uses to get nothing but the header of a document. When used on an FTP or FILE file, curl displays the file size and last modification time only.

-u user:password --user user:password

The username and password to use for server authentication. Overrides -n, –netrc and –netrc-optional. If you just give the user name (without entering a colon) curl will prompt for a password. If you use an SSPI-enabled curl binary and do NTLM authentication, you can force curl to pick up the username and password from your environment by specifying a single colon with this option: “-u :”. If this option is used several times, the last one will be used.

-x host:port -x [protocol://][user:password@]proxyhost[:port] --proxy [protocol://][user:password@]proxyhost[:port]

Use the specified HTTP proxy. If the port number is not specified, it is assumed at port 1080.

-H "name: value" --header "name: value"

Add Header when getting a web page. You may specify any number of extra headers.

-H “name:”
–header “name:”

Remove Header, remove an internal header.

-L, –location Follow redirects if the server reports that the requested page has moved (indicated with a Location: header and a 3XX response code)

-v, –verbose Make more verbose/talkative. Mostly useful for debugging.

Examples

(1) If request method is not specified, curl will send GET request. e.g.

$ curl https://api.github.com/users/caspyin

(2) Includes response HTTP-Header information in the output

$ curl --include https://api.github.com/users/caspyin

(3) Pass HTTP Authentication in cURL

$ curl -u username:password URL

$ curl --user "caspyin:PASSWD" https://api.github.com/gists/starred
$ curl --user "caspyin:PASSWD" https://api.github.com/users/caspyin

Note: By default curl uses Basic HTTP Authentication. We can specify other authentication method using –ntlm | –digest.

(4) Retrieve a web page and save to a file.

$ curl http://www.gnu.org/download/gccxxx.tar.xz > gcc.xz

or

$ curl http://www.gnu.org/download/gccxxx.tar.xz -o gcc.xz

or

$ curl http://www.gnu.org/download/gccxxx.tar.xz -O

(5) To retrieve a web page, or its redirected target

$ curl www.tutorialspoint.com/unix/ 
$ curl www.tutorialspoint.com/unix/ -L

(6) Use Proxy to Download a File

$ curl -x proxysever.test.com:3128 http://google.co.in

(7) Continue/Resume a Previous Download

Start a big download using curl, and press Ctrl-C to stop it in between the download.

$ curl -O http://www.gnu.org/software/gettext/manual/gettext.html
##############             20.1%

Now the above download was stopped at 20.1%. Using “curl -C -“, we can continue the download from where it left off earlier. Now the download continues from 20.1%.

$ curl -C - -O http://www.gnu.org/software/gettext/manual/gettext.html
###############            21.1%

(8) List/Download using Ranges

cURL supports ranges to be given in the URL. When a range is given, files matching within the range will be downloaded. It will be helpful to download packages from the FTP mirror sites.

$ curl   ftp://ftp.uk.debian.org/debian/pool/main/[a-z]/

(9) POST

$ curl --user "caspyin" -X POST --data '{"description":"Created via API","public":"true","files":{"file1.txt":{"content":"Demo"}}' https://api.github.com/gists

since –data implies POST there is no need to specify the –request flag

$ curl --user "caspyin" --data '{"description":"Created via API","public":"true","files":{"file1.txt":{"content":"Demo"}}' https://api.github.com/gists

Tell curl to read from a file (@) to POST data

$ curl --user "caspyin" --data @data.txt https://api.github.com/gists 

Or read from STDIN (@-)

$ echo '{"text": "Hello world!"}' | curl -d @- https://api.github.com/markdown

Url-encoded

$ curl --data "brand=nike&color=red&size=11" $URL

Multipart

$ curl -F "image=@nikes.png" -F "brand=nike" -F "color=red" -F "size=11" $URL

Change the name field of a file upload part (nikes.png) by setting filename=NEW_NAME”

$ curl -F "image=@nikes.png;filename=shoes.png" -F "brand=nike" -F "color=red" -F "size=11" $URL

Specify Content-Type by using type=:

$ curl -F "image=@nikes.png;type=image/png" -F "brand=nike" -F "color=red" -F "size=11" $URL

More examples:

With fields:

$ curl --data "param1=value1&param2=value2" https://example.com/resource.cgi

Multipart:

$ curl --form "fileupload=@my-file.txt" https://example.com/resource.cgi

Multipart with fields and a filename:

$ curl --form "fileupload=@my-file.txt;filename=desired-filename.txt" --form param1=value1 --form param2=value2 https://example.com/resource.cgi

Specifying the Content-Type while POST a file:

For XML

$ curl -X POST -d @filename.txt http://example.com/path/to/resource --header "Content-Type:text/xml"

For JSON

$ curl -X POST -d @filename.txt http://example.com/path/to/resource --header "Content-Type:application/json"

(10) Headers

Often when POSTing data you’ll need to add headers for things like auth tokens or setting the content type. You can set a header using -H.

$ curl -H "Content-Type: application/json" -H "authToken: 349ab29a-xtab-423b-a5hc-5623bc39b8c8" --data '{}' https://api.example.com/endpoint

Reference
https://www.tutorialspoint.com/unix_commands/curl.htm