curl.md

@@ -1,6 +1,8 @@
 # Curl useful commands
 
-Simplest way to `GET` data
+This is a cheatsheet taken from the [how to curl better](https://conf.tube/w/qNm5tMUqHnGZPbtCyhxnUd).
+
+## Simplest way to `GET` data
 
 ```bash
 $ curl example.com
@@ -29,3 +31,115 @@ Please follow redirects
 ```bash
 $ curl -I -L https://example.com/redirected
 ```
+
+## URL Globbing
+
+```bash
+$ curl https://example.com/[1-9].html
+```
+
+```bash
+$ curl https://example.com/[01-99].html
+```
+
+```bash
+$ curl https://example.com/[a-z].html
+```
+
+Provide a step when globbing
+
+```bash
+$ curl https://example.com/[1-9:2].html
+```
+
+```bash
+$ curl https://example.com/[a-z:3].html
+```
+
+Save the matching files to output with `#1` -> [1-9] parameter
+
+```bash
+$ curl https://example.com/[1-9].html -o save_#1.html
+```
+
+Comma separated strings
+
+```bash
+$ curl https://example.com/{ham,cheese,pineapple}.jpg -o hawaii_#1.jpg
+```
+
+Combine everything in one line
+
+```bash
+$ curl https://example.com/issue[1996-1999]/vol[1-4]/part{a,b,c}.html
+```
+
+## Verbose shows more from under the hood
+
+```bash
+$ curl -v https://example.com/ -o /dev/null
+```
+
+## Pass in custom headers
+
+```bash
+$ curl https://example.com/ -H "User-Agent: Some Silly Agent"
+$ curl https://example.com/ -H "Magic: disc0"
+$ curl https://example.com/ -H "User-Agent:"
+$ curl https://example.com/ -H "User-Agent;"
+```
+
+## Methods
+
+POST some basic data to the remote
+
+```bash
+$ curl -d name=Daniel -i https://example.com/receiver
+```
+
+POST a file
+
+```bash
+$ curl -d @file https://example.com/receiver -o saved
+# Post a standard input
+$ ls -l | curl -d @- https://example.com/receiver -o saved
+# Post as binaries
+$ ls -l | curl --data-binary @- https://example.com/receiver
+# Post json as binary
+$ curl --data-binary @file.json -H "Content-Type: application/json" https://example.com
+```
+
+PUT a file
+
+```bash
+$ curl -T localfile -i https://example.com/remote_name
+```
+
+Change the method string
+
+```bash
+# Use -X if you want a different menthod than curl would use
+curl -T localfile -X SWOOSH https://example.com/remote_name -o save
+```
+
+## Cookies
+
+Save cookies from site
+
+```bash
+$ curl -c cookiejar.txt https://example.com/
+```
+
+Send cookies to the server
+
+```bash
+$ curl -b cookiejar.txt https://example.com/
+```
+
+Cookies in a login
+
+```bash
+$ curl -b cookiejar.txt -c cookiejar.txt https://example.com/login -d user=daniel -d password=1234
+# Request data as a logged in user
+$ curl -b cookiejar.txt -c cookiejar.txt https://example.com/profile
+```

wget.md

@@ -0,0 +1,88 @@
+Wget recursive download options
+=====
+
+``` bash
+wget --recursive -np -nc -nH --cut-dirs=4 --random-wait --wait 1 -e robots=off https://site.example/aaa/bbb/ccc/ddd/
+```
+
+This downloads the files to whatever directory you ran the command in. To use Wget to recursively download using FTP, simply change `https://` to `ftp://` using the FTP directory.
+
+``` bash
+--recursive
+```
+
+download recursively (and place in recursive folders on your PC)
+
+``` bash
+--recursive --level=1
+```
+
+recurse but --level=1 don’t go below specified directory
+
+``` bash
+-Q 1g
+```
+
+total overall download --quota option, for example to stop downloading after 1 GB has been downloaded altogether
+
+``` bash
+-np
+```
+
+Never get parent directories (sometimes a site will link upwards)
+
+``` bash
+-nc
+```
+
+no clobber – don’t re-download files you already have
+
+``` bash
+-nd
+```
+
+no directory structure on download (put all files in one directory commanded by -P)
+
+``` bash
+-nH
+```
+
+don’t put vestigial site name directories on your PC
+
+``` bash
+-A
+```
+
+only accept files matching globbed pattern
+
+``` bash
+--cut-dirs=4
+```
+
+don’t put a vestigial hierarchy of directories above the desired directory on your PC. Set the number equal to the number of directories on server (here aaa/bbb/ccc/ddd is four)
+	
+``` bash
+-e robots=off
+```
+
+Many sites will block robots from mindlessly consuming huge amounts of data. Here we override this setting telling Apache that we’re (somewhat) human.
+	
+``` bash
+--random-wait
+```
+
+To avoid excessive download requests (that can get you auto-banned from downloading) we politely wait in-between file downloads
+	
+``` bash
+--wait 1
+```
+
+making the random wait time average to about 1 second before starting to download the next file. This helps avoid anti-leeching measures.
+
+
+## References:
+
++ [scivision](https://www.scivision.dev/wget-recursive-download/)
++ [wget recursive download](https://www.gnu.org/software/wget/manual/html_node/Recursive-Download.html#Recursive-Download)
++ [comparison table](https://curl.haxx.se/docs/comparison-table.html)
++ [recursive retrieval options](https://www.gnu.org/software/wget/manual/html_node/Recursive-Retrieval-Options.html)