README.adoc 54 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362363364365366367368369370371372373374375376377378379380381382383384385386387388389390391392393394395396397398399400401402403404405406407408409410411412413414415416417418419420421422423424425426427428429430431432433434435436437438439440441442443444445446447448449450451452453454455456457458459460461462463464465466467468469470471472473474475476477478479480481482483484485486487488489490491492493494495496497498499500501502503504505506507508509510511512513514515516517518519520521522523524525526527528529530531532533534535536537538539540541542543544545546547548549550551552553554555556557558559560561562563564565566567568569570571572573574575576577578579580581582583584585586587588589590591592593594595596597598599600601602603604605606607608609610611612613614615616617618619620621622623624625626627628629630631632633634635636637638639640641642643644645646647648649650651652653654655656657658659660661662663664665666667668669670671672673674675676677678679680681682683684685686687688689690691692693694695696697698699700701702703704705706707708709710711712713714715716717718719720721722723724725726727728729730731732733734735736737738739740741742743744745746747748749750751752753754755756757758759760761762763764765766767768769770771772773774775776777778779780781782783784785786787788789790791792793794795796797798799800801802803804805806807808809810811812813814815816817818819820821822823824825826827828829830831832833834835836837838839840841842843844845846847848849850851852853854855856857858859860861862863864865866867868869870871872873874875876877878879880881882883884885886887888889890891892893894895896897898899900901902903904905906907908909910911912913914915916917918919920921922923924925926927928929930931932933934935936937938939940941942943944945946947948949950951952953954955956957958959960961962963964965966967968969970971972973974975976977978979980981982983984985986987988989990991992993994995996997998999100010011002100310041005100610071008100910101011101210131014101510161017101810191020102110221023102410251026102710281029103010311032103310341035103610371038103910401041104210431044104510461047104810491050105110521053105410551056105710581059106010611062106310641065106610671068106910701071107210731074107510761077107810791080108110821083108410851086108710881089109010911092109310941095109610971098109911001101110211031104110511061107110811091110111111121113111411151116111711181119112011211122112311241125112611271128112911301131113211331134113511361137113811391140114111421143114411451146114711481149115011511152115311541155115611571158115911601161116211631164116511661167116811691170117111721173117411751176117711781179118011811182118311841185118611871188118911901191119211931194119511961197119811991200120112021203120412051206120712081209121012111212121312141215121612171218121912201221122212231224122512261227122812291230123112321233123412351236123712381239124012411242124312441245124612471248124912501251125212531254125512561257125812591260126112621263126412651266126712681269127012711272127312741275127612771278127912801281128212831284128512861287128812891290129112921293129412951296129712981299130013011302130313041305130613071308130913101311131213131314131513161317131813191320132113221323132413251326132713281329133013311332133313341335133613371338133913401341134213431344134513461347134813491350135113521353135413551356135713581359136013611362136313641365136613671368136913701371137213731374137513761377137813791380138113821383138413851386138713881389139013911392139313941395139613971398139914001401140214031404140514061407140814091410141114121413141414151416141714181419142014211422142314241425142614271428142914301431143214331434143514361437143814391440144114421443144414451446144714481449145014511452145314541455145614571458145914601461146214631464146514661467146814691470147114721473147414751476147714781479148014811482148314841485148614871488148914901491149214931494149514961497149814991500150115021503150415051506150715081509151015111512151315141515151615171518151915201521152215231524152515261527152815291530153115321533153415351536153715381539154015411542154315441545154615471548154915501551155215531554155515561557155815591560156115621563156415651566156715681569157015711572157315741575157615771578157915801581158215831584158515861587158815891590159115921593159415951596159715981599160016011602160316041605160616071608160916101611161216131614161516161617161816191620162116221623162416251626162716281629163016311632163316341635163616371638163916401641164216431644164516461647164816491650165116521653165416551656165716581659166016611662166316641665166616671668166916701671167216731674167516761677167816791680168116821683168416851686168716881689169016911692169316941695169616971698169917001701170217031704170517061707170817091710171117121713171417151716171717181719172017211722172317241725172617271728172917301731173217331734173517361737173817391740174117421743174417451746174717481749175017511752175317541755175617571758175917601761176217631764176517661767176817691770177117721773177417751776177717781779178017811782178317841785178617871788178917901791179217931794179517961797179817991800180118021803180418051806180718081809181018111812181318141815181618171818181918201821182218231824182518261827182818291830183118321833183418351836183718381839184018411842184318441845184618471848184918501851185218531854185518561857185818591860186118621863186418651866186718681869187018711872187318741875187618771878187918801881188218831884188518861887188818891890189118921893189418951896189718981899190019011902190319041905190619071908190919101911191219131914191519161917191819191920192119221923192419251926192719281929193019311932193319341935193619371938193919401941194219431944194519461947194819491950195119521953195419551956195719581959196019611962196319641965196619671968196919701971197219731974197519761977197819791980198119821983198419851986198719881989199019911992199319941995199619971998199920002001200220032004200520062007200820092010201120122013201420152016201720182019
  1. = x86 Bare Metal Examples
  2. :idprefix:
  3. :idseparator: -
  4. :sectanchors:
  5. :sectlinks:
  6. :sectnumlevels: 6
  7. :sectnums:
  8. :toc: macro
  9. :toclevels: 6
  10. :toc-title:
  11. Dozens of minimal operating systems to learn x86 system programming. Tested on Ubuntu 17.10 host. Userland cheat at: https://github.com/cirosantilli/x86-assembly-cheat
  12. toc::[]
  13. == Getting started
  14. First read this introduction: https://stackoverflow.com/questions/22054578/how-to-run-a-program-without-an-operating-system/32483545#32483545
  15. On Ubuntu:
  16. ....
  17. ./configure
  18. make
  19. ....
  20. Each `.S` file on the top-level is an operating system! It gets compiled to a corresponding `.img` file.
  21. Run the default OS on QEMU:
  22. ....
  23. ./run
  24. ....
  25. Run a given OS:
  26. ....
  27. ./run min
  28. ./run bios_one_char
  29. ....
  30. Extensions are ignored for perfect tab completion, so all the following are equivalent:
  31. ....
  32. ./run min
  33. ./run min.
  34. ./run min.S
  35. ./run min.img
  36. ....
  37. Use Bochs instead of QEMU:
  38. ....
  39. ./run bios_hello_world bochs
  40. ....
  41. Then on the terminal start the simulation with:
  42. ....
  43. c
  44. ....
  45. https://stackoverflow.com/questions/6142925/how-can-i-use-bochs-to-run-assembly-code/32871939#32871939
  46. === Getting started with real hardware
  47. Insert an USB, determine its device (`/dev/sdX`) with:
  48. ....
  49. sudo lsblk
  50. sudo fdisk -l
  51. ....
  52. Pick the `.img` file that you wan to run and:
  53. ....
  54. sudo dd if=bios_hello_world.img of=/dev/sdX
  55. ....
  56. Then:
  57. * insert the USB in a computer
  58. * during boot, hit some special hardware dependant key, usually F12, Esc
  59. * choose to boot from the USB
  60. When you are done, just hit the power button to shutdown.
  61. See: <<test-hardware>>
  62. ==== Getting started with the big image
  63. Create a `big.img` that contains all examples that can be booted from GRUB:
  64. ....
  65. make big.img
  66. ....
  67. Now if you do:
  68. ....
  69. sudo dd if=big.img of=/dev/sdX
  70. ....
  71. you can test several examples with a single USB burn, which is much faster.
  72. You can also try out the big image on QEMU for fun with:
  73. ....
  74. qemu-system-i386 -hda big.img
  75. ....
  76. You will also want to change the boot order to put the USB first from the F12 BIOS menu. This way you don't have to hit F12 like a madman every time.
  77. TODO: boot sectors that load STAGE2 are not working with the big image chainloader. TODO why?
  78. === Getting started with Docker
  79. If you don't have an Ubuntu box, this is an easy alternative:
  80. ....
  81. sudo docker run -it --net=host ubuntu:14.04 bash
  82. ....
  83. Then proceed normally in the guest: install packages, and build:
  84. ....
  85. apt-get update
  86. apt-get install git
  87. git clone https://github.com/cirosantilli/x86-bare-metal-examples
  88. cd x86-bare-metal-examples
  89. ./configure
  90. make
  91. ....
  92. To overcome the lack of GUI, we can use QEMU's VNC implementation instead of the default SDL, which is visible on the host due to `--net=host`:
  93. ....
  94. qemu-system-i386 -hda main.img -vnc :0
  95. ....
  96. and then on host:
  97. ....
  98. sudo apt-get install vinagre
  99. vinagre localhost:5900
  100. ....
  101. === GDB step debug
  102. TODO get it working nicely:
  103. ....
  104. ./run bios_hello_world debug
  105. ....
  106. This will only cover specifics, you have to know GDB debugging already.
  107. How to have debug symbols: https://stackoverflow.com/questions/32955887/how-to-disassemble-16-bit-x86-boot-sector-code-in-gdb-with-x-i-pc-it-gets-tr/32960272#32960272 TODO implement here. Needs to point GDB to an ELF file in addition to the remote listen.
  108. How to step over `int` calls: http://stackoverflow.com/questions/24491516/how-to-step-over-interrupt-calls-when-debugging-a-bootloader-bios-with-gdb-and-q
  109. Single stepping until a given opcode can be helpful sometimes: https://stackoverflow.com/questions/14031930/break-on-instruction-with-specific-opcode-in-gdb/31249378#31249378
  110. TODO: detect if we are on 16 or 32 bit automatically from control registers. Now I'm using 2 functions `16` and `32` to switch manually, but that sucks. The problem is that it's not possible to read them directly: http://stackoverflow.com/a/31340294/895245 If we had `cr0`, it would be easy to do with an `if cr0 & 1` inside a hook-stop.
  111. TODO: Take segmentation offsets into account: http://stackoverflow.com/questions/10354063/how-to-use-a-logical-address-in-gdb
  112. == Minimal examples
  113. These are the first ones you should look at.
  114. [[printf]]
  115. === Create a minimal image with printf
  116. ....
  117. make -C printf run
  118. ....
  119. Outcome: QEMU window opens up, prints a few boot messages, and hangs.
  120. Our program itself does not print anything to the screen itself, just makes the CPU halt.
  121. This example is generated with `printf` byte by byte: you can't get more minimal than this!
  122. It basically consists of:
  123. * byte 0: a `hlt` instruction
  124. * bytes 510 and 511: mandatory magic bytes `0xAA55`, which are required for BIOS to consider our disk.
  125. === Minimal GAS example
  126. ....
  127. ./run min
  128. ....
  129. Outcome: QEMU window opens up, prints a few firmware messages, and hangs.
  130. === BIOS hello world
  131. ....
  132. ./run bios_hello_world
  133. ....
  134. Outcome:
  135. ....
  136. hello world
  137. ....
  138. shows after the firmware messages.
  139. ==== Linker script
  140. This hello world, and most of our OSes use the linker script link:linker.ld[].
  141. This critical file determines the memory layout of our assembly, take some time to read the comments in that file and familiarize yourself with it.
  142. The Linux kernel also uses linker scripts to setup its image memory layout, see for example: https://github.com/torvalds/linux/blob/v4.2/arch/x86/boot/setup.ld
  143. === No linker script
  144. ....
  145. make -C no-linker-script run
  146. ....
  147. Outcome:
  148. ....
  149. hello world
  150. ....
  151. Uses the default host `ld` script, not an explicit one set with `-T`. Uses:
  152. * `-tText`
  153. * `.org` inside each assembly file
  154. * `_start` must be present to avoid a warning, since the default linker script expects it
  155. Less stable, but more convenient for quick and dirty tests.
  156. == BIOS
  157. https://en.wikipedia.org/wiki/BIOS
  158. http://wiki.osdev.org/BIOS
  159. The BIOS is one of the most well known firmwares in existence.
  160. A firmware is a software a software that:
  161. * runs before the OS / bootloader to do very low level setup
  162. * usually closed source, provided by the vendor, and interacts with undocumented hardware APIs
  163. * offers an API to the OS / bootloader, that allows you to do things like quick and dirty IO
  164. * undistinguishable from an OS, except that is it usually smaller
  165. BIOS is old, non-standardized, x86 omnipresent and limited.
  166. <<uefi>> is the shiny new overbloated thing.
  167. If you are making a serious OS, use it as little as possible.
  168. Can only be used in real mode.
  169. BIOS functions are all accessed through the `int` instruction:
  170. ....
  171. mov <function-id>, %ah
  172. int <interrupt-id>
  173. ....
  174. Function arguments are stored in other registers.
  175. The interrupt IDs are traditionally in hex as:
  176. ....
  177. 10h
  178. ....
  179. which is the same as `0x10`.
  180. Each `interrupt-id` groups multiple functions with similar functions, e.g. `10h` groups functions with video related functionality.
  181. === BIOS documentation
  182. Does any official documentation or standardization exist?
  183. * https://en.wikipedia.org/wiki/BIOS_interrupt_call#Interrupt_table
  184. * http://www.ctyme.com/intr/int.htm Ralf Brown's Interrupt List. Everyone says that this is the ultimate unofficial compilation.
  185. * https://en.wikipedia.org/wiki/INT_10H good quick summary
  186. * http://www.scs.stanford.edu/nyu/04fa/lab/specsbbs101.pdf says little about interrupts, I don't understand it's scope.
  187. === BIOS examples
  188. Print a single `@` character:
  189. ....
  190. ./run bios_putc
  191. ....
  192. Print a newline with bios:
  193. ....
  194. ./run bios_newline
  195. ....
  196. Outcome:
  197. ....
  198. hello
  199. world
  200. ....
  201. Carriage returns are needed just like in old days:
  202. ....
  203. ./run bios_carriage_return
  204. ....
  205. Outcome:
  206. ....
  207. hello
  208. world
  209. ....
  210. Change the current cursor position:
  211. ....
  212. ./run bios_cursor_position
  213. ....
  214. Outcome:
  215. ....
  216. cb
  217. ....
  218. ==== BIOS color
  219. Color codes: https://en.wikipedia.org/wiki/BIOS_color_attributes
  220. Write a character N times with given color:
  221. ....
  222. ./run bios_color
  223. ....
  224. Outcome:
  225. ....
  226. bcd
  227. ....
  228. where:
  229. * `b` and `c` have red foreground, and green background
  230. * `d` has the default color (gray on black)
  231. Change the background color to red for the entire screen and print an `a` character:
  232. ....
  233. ./run bios_background
  234. ....
  235. ==== BIOS scroll
  236. Scroll the screen:
  237. ....
  238. ./run bios_scroll
  239. ....
  240. Outcome:
  241. ....
  242. a
  243. c
  244. GG
  245. d
  246. ....
  247. where `G` are empty green squares.
  248. How it works:
  249. Before scroll:
  250. ....
  251. a
  252. b
  253. c
  254. d
  255. ....
  256. We then choose to act on the rectangle with corners (1, 1) and (2, 2) given by `cx` and `dx`:
  257. ....
  258. a
  259. XX
  260. YY
  261. d
  262. ....
  263. and scroll that rectangle up by one line.
  264. `Y` is then filled with the fill color green
  265. ===== BIOS clear screen
  266. Subset of scroll:
  267. ....
  268. ./run bios_clear_screen
  269. ....
  270. Outcome:
  271. ....
  272. b
  273. ....
  274. on red foreground, and the entire screen in green background, without any initial SeaBIOS messages.
  275. ==== BIOS draw pixel
  276. Make the pixel at position (1, 1) clear red color (0Ch) in <<video-mode-13h>>:
  277. ....
  278. ./run bios_pixel
  279. ....
  280. You may have to look a bit hard to see it.
  281. Draw a line of such pixels:
  282. ....
  283. ./run bios_pixel_line
  284. ....
  285. Advanced graphics!
  286. ==== BIOS keyboard
  287. https://stackoverflow.com/questions/4113250/how-to-handle-keyboard-in-real-mode-through-bios-interrupts/32682518#32682518
  288. Get one character from the user via the keyboard, increment it by one, and print it to the screen, then halt:
  289. ....
  290. ./run bios_keyboard
  291. ....
  292. Type a bunch of characters and see them appear on the screen:
  293. ....
  294. ./run bios_keyboard_loop
  295. ....
  296. Do try `Ctrl-key` combinations.
  297. ==== BIOS disk load
  298. Load a stage 2 from disk with `int 13h` and run it:
  299. ....
  300. ./run bios_disk_load
  301. ....
  302. Outcome:
  303. ....
  304. a
  305. ....
  306. and this character was printed from stage 2.
  307. Load two sectors instead of just one:
  308. ....
  309. ./run bios_disk_load2
  310. ....
  311. Outcome:
  312. ....
  313. ab
  314. ....
  315. where `a` was printed from code on the first block, and `b` from code on the second block.
  316. This shows that each sector is 512 bytes long.
  317. Grub 2.0 makes several calls to it under `grub-core/boot/i386/pc`.
  318. TODO: not working on Bochs: `BOUND_GdMa: fails bounds test`.
  319. But it does work on QEMU and <<thinkpad-t400>>.
  320. Bibliography:
  321. * https://en.wikipedia.org/wiki/INT_13H
  322. * http://wiki.osdev.org/ATA_in_x86_RealMode_%28BIOS%29
  323. * https://thiscouldbebetter.wordpress.com/2011/03/15/creating-a-bootable-program-in-assembly-language/
  324. * http://stackoverflow.com/questions/19381434/cannot-read-disk-sectors-in-assembly-language
  325. * http://stackoverflow.com/questions/15497842/read-a-sector-from-hard-drive-with-int-13h
  326. ==== BIOS detect memory
  327. An yet failed TODO attempt at detecting how big our memory is with `int 15h`:
  328. ....
  329. ./run bios_detect_memory
  330. ....
  331. Seems to output trash currently.
  332. http://wiki.osdev.org/Detecting_Memory_%28x86%29
  333. This is important in particular so that you can start your stack there when you enter <<protected-mode>>, since the stack grows down.
  334. In 16-bit mode, it does not matter much, since most modern machines have all addressable memory there, but in 32-bit protected it does, as our emulator usually does not have all 4Gb. And of course, 64-bit RAM is currently larger than the total RAM in the world.
  335. `int 15` returns a list: each time you call it a new memory region is returned.
  336. The format is not too complicated, and documented at: http://wiki.osdev.org/Detecting_Memory_%28x86%29#Detecting_Upper_Memory
  337. * 8 bytes: base address of region.
  338. * 8 bytes: length of region.
  339. * 4 bytes: type or region. 1 for usable RAM.
  340. * 4 bytes: some ACPI stuff that no one uses?
  341. ===== Low vs high memory
  342. TODO
  343. `int 15h` can detect low or high memory. How are they different?
  344. ==== BIOS sleep
  345. Count to infinity, sleep one second between each count:
  346. ....
  347. ./run bios_sleep
  348. ....
  349. https://stackoverflow.com/questions/9971405/how-to-display-a-number-on-the-screen-and-and-sleep-for-one-second-with-dos-x86/9973442#9973442
  350. Polls time counter that BIOS keeps up to date at `0x046C` with frequency 18.2Hz eighteen times.
  351. ==== BIOS initial state
  352. Check the initial state the firmware leaves us in:
  353. ....
  354. ./run bios_initial_state
  355. ....
  356. Prints the contents of several registers.
  357. `dx` seems to be like the only interesting regular register: the firmware stores the value of the current disk number to help with `int 15h` there. Thus it usually contains `0x80`.
  358. === dmidecode
  359. Get BIOS information.
  360. * http://stackoverflow.com/questions/20604644/how-to-check-the-bios-version-or-name-in-linux-through-command-prompt
  361. * https://en.wikipedia.org/wiki/System_Management_BIOS SMBIOS
  362. Try it on host:
  363. ....
  364. sudo dmidecode
  365. ....
  366. Standardized by: https://en.wikipedia.org/wiki/Distributed_Management_Task_Force
  367. TODO: how is it obtained at the low level?
  368. === SeaBIOS
  369. http://www.seabios.org/SeaBIOS
  370. Open source x86 BIOS implementation.
  371. Default BIOS for QEMU and KVM.
  372. == Modes of operation
  373. The x86 processor has a few modes, which have huge impact on how the processor works.
  374. Covered on the <<intel-manual>> Volume 3. Specially useful is the "Figure 2-3. Transitions Among the Processor’s Operating Modes" diagram.
  375. The modes are:
  376. * Real-address, usually known just as "real mode"
  377. * Protected
  378. * System management
  379. * IA-32e. Has two sub modes:
  380. ** Compatibility
  381. ** 64-bit
  382. * Virtual-8086 Mode
  383. Transition tables:
  384. ....
  385. (all modes)
  386. |
  387. | Reset
  388. |
  389. v
  390. +---------------------+
  391. | Real address (PE=0) |
  392. +---------------------+
  393. ^
  394. |
  395. | PE
  396. |
  397. v
  398. +------------------------+
  399. | Protected (PE=1, VM=0) |
  400. +------------------------+
  401. ^ ^
  402. | |
  403. | | VM
  404. | |
  405. v v
  406. +--------------+ +---------------------+
  407. | IA-32e | | Virtual-8086 (VM=1) |
  408. +--------------+ +---------------------+
  409. ....
  410. and:
  411. ....
  412. +------------------------+
  413. | System management mode |
  414. +------------------------+
  415. | ^
  416. | |
  417. | RSM | SMI#
  418. | |
  419. v |
  420. (All other modes)
  421. ....
  422. The IA-32e transition is trickier, but clearly described on the <<intel-manual>> Volume 3 - 9.8.5 "Initializing IA-32e Mode":
  423. ____
  424. Operating systems should follow this sequence to initialize IA-32e mode:
  425. 1. Starting from protected mode, disable paging by setting `CR0.PG = 0`. Use the `MOV CR0` instruction to disable paging (the instruction must be located in an identity-mapped page).
  426. 2. Enable physical-address extensions (PAE) by setting CR4.`PAE = 1`. Failure to enable PAE will result in a `#GP` fault when an attempt is made to initialize IA-32e mode.
  427. 3. Load `CR3` with the physical base address of the Level 4 page map table (PML4).
  428. 4. Enable IA-32e mode by setting `IA32_EFER.LME = 1`.
  429. 5. Enable paging by setting `CR0.PG = 1`. This causes the processor to set the `IA32_EFER.LMA` bit to 1. The `MOV CR0` instruction that enables paging and the following instructions must be located in an identity-mapped page (until such time that a branch to non-identity mapped pages can be effected).
  430. ____
  431. === Legacy modes
  432. The term defined in the <<intel-manual>> Volume 3 - CHAPTER 2 "SYSTEM ARCHITECTURE OVERVIEW":
  433. ____
  434. Real mode, protected mode, virtual 8086 mode, and system management mode. These are sometimes referred to as legacy modes.
  435. ____
  436. In other words: anything except IA-32e and System management mode.
  437. This further suggests that real, protected and virtual mode are not the main intended modes of operation.
  438. === Real mode
  439. http://wiki.osdev.org/Real_Mode
  440. The CPU starts in this mode after power up.
  441. All our <<bios>> examples are in real mode.
  442. It is possible to use 32-bit registers in this mode with the "Operand Size Override Prefix" `0x66`.
  443. TODO is it possible to access memory above 1M like this:
  444. ....
  445. mov $1, 0xF0000000
  446. mov $1, (%eax)
  447. ....
  448. http://stackoverflow.com/questions/6917503/is-it-possible-to-use-32-bits-registers-instructions-in-real-mode
  449. ==== Real mode segmentation
  450. ....
  451. ./run real_segmentation
  452. ....
  453. Outcome:
  454. ....
  455. AAAAAA
  456. ....
  457. We access the character `A` with segments in 6 different ways:
  458. * `ds`, with explicit and implicit segment syntax
  459. * `es`, `fs`, `gs`, `ss`
  460. Segment registers modify the addresses that instructions actually use as:
  461. ....
  462. <segment> * 16 + <original-address>
  463. ....
  464. This implies that:
  465. * 20 bits of memory (1MB) instead of the 16 bits (256kB) that normally fits into registers. E.g., to address:
  466. +
  467. ....
  468. 0x84000
  469. ....
  470. +
  471. we can use:
  472. +
  473. ....
  474. 0x8000 (segment)
  475. 0x 4000 (address)
  476. -------
  477. 0x84000
  478. ....
  479. * most addresses can be encoded in multiple ways, e.g.:
  480. +
  481. ....
  482. 0x100
  483. ....
  484. +
  485. can be encoded as either of:
  486. +
  487. ** segment = `0x10`, address = `0`
  488. ** segment = `0`, address = `0x100`
  489. ** segment = `0x1`, address = `0xF0`
  490. `fs` and `gs` are general purpose: they are not affected implicitly by any instructions. All others will be further exemplified.
  491. ===== CS
  492. https://stackoverflow.com/questions/17777146/what-is-the-purpose-of-cs-and-ip-registers-in-intel-8086-assembly/33177253#33177253
  493. Affects the code address pointer:
  494. ....
  495. ./run cs
  496. ....
  497. Outcome:
  498. ....
  499. 00
  500. 01
  501. 02
  502. ....
  503. `CS` is set with the `ljmp` instruction, and we use it to skip `.skip` zero gaps in the code.
  504. ===== SS
  505. ....
  506. ./run ss
  507. ....
  508. Outcome:
  509. ....
  510. 0102
  511. ....
  512. The second byte is 16 bytes after the first, and is accessed with `SP = 1`.
  513. `SS` affects instructions that use `SP` such as `PUSH` and `POP`: those will actually use `16 * SS + SP` as the actual address.
  514. ===== ES
  515. TODO: this does seem to have special properties as used by string instructions.
  516. ===== Segment register encoding
  517. ....
  518. objdump -D -b binary -m i8086 segment_registers.img
  519. ....
  520. shows that non `ds` encodings are achieved through a prefix:
  521. ....
  522. 20: a0 63 7c mov 0x7c63,%al
  523. 34: 26 a0 63 7c mov %es:0x7c63,%al
  524. 40: 64 a0 63 7c mov %fs:0x7c63,%al
  525. 4c: 65 a0 63 7c mov %gs:0x7c63,%al
  526. 58: 36 a0 63 7c mov %ss:0x7c63,%al
  527. ....
  528. This makes `ds` the most efficient one for data access, and thus a good default.
  529. ==== Interrupts
  530. ....
  531. ./run interrupt
  532. ....
  533. Outcome:
  534. ....
  535. ab
  536. ....
  537. It works like this:
  538. * print `a` an interrupt handler `0`
  539. * jump back to main code
  540. * print `b`
  541. TODO: is STI not needed because this interrupt is not maskable?
  542. Same with interrupt handler `1`:
  543. ....
  544. ./run interrupt1
  545. ....
  546. TODO understand: attempt to create an infinite loop that calls the interrupt from the handler:
  547. ....
  548. ./run interrupt_loop
  549. ....
  550. QEMU exits with:
  551. ....
  552. Trying to execute code outside RAM or ROM at 0x000a0000
  553. ....
  554. Handle a division by zero:
  555. ....
  556. ./run interrupt_zero_divide
  557. ....
  558. TODO understand:
  559. * expected outcome: prints values from 0 to `0xFFFF` in an infinite loop.
  560. * actual outcome: stops at `0081`
  561. Apparently when there is an exception, `iret` jumps back to the line that threw the exception itself, not the one after, which leads to the loop:
  562. * https://stackoverflow.com/questions/33029457/what-to-do-in-interrupt-handler-for-divide-by-zero
  563. * https://stackoverflow.com/questions/9151429/os-development-how-to-avoid-an-infinite-loop-after-an-exception-routine
  564. But then why does it stop at `0081`? And if we set the initial value to `0x0090`, it just runs once.
  565. ===== int
  566. * long jumps to the CS : IP found in the corresponding interrupt vector.
  567. * pushes EFLAGS to let them be restored by iret?
  568. ===== iret
  569. Jumps back to the next instruction to be executed before the interrupt came in.
  570. Restores EFLAGS and other registers TODO which?
  571. Vs `jmp`: http://stackoverflow.com/questions/10462884/must-iret-be-used-when-returning-from-an-interrupt
  572. ===== Interrupt service routines
  573. Fancy name for the handler: http://wiki.osdev.org/Interrupt_Service_Routines
  574. ===== IVT
  575. Interrupt vector table: https://wiki.osdev.org/IVT
  576. The real mode in-memory table that stores the address for the handler for each interrupt.
  577. In <<protected-mode>>, the analogous structure is the <<idt>>.
  578. The base address is set in the interrupt descriptor table register (IDTR), which can be modified with the <<lidt>> instruction.
  579. The default address is `0x0`.
  580. The format of the table is:
  581. ....
  582. IDTR -> +-----------------------+
  583. 0 |Address (2 bytes) |
  584. 2 |Code segment (2 bytes) |
  585. +-----------------------+
  586. +-----------------------+
  587. 4 ----> |Address (2 bytes) |
  588. 6 |Code segment (2 bytes) |
  589. +-----------------------+
  590. +-----------------------+
  591. 8 ----> |Address (2 bytes) |
  592. A |Code segment (2 bytes) |
  593. +-----------------------+
  594. ... ...
  595. ....
  596. ====== lidt
  597. Set the value of the IDTR, and therefore set the base address of the <<ivt>>:
  598. ....
  599. ./run lidt
  600. ./run lidt2
  601. ./run lidt0
  602. ....
  603. TODO not working.
  604. Expected outcome:
  605. ....
  606. ab
  607. ....
  608. Actual outcome: infinite reboot loop.
  609. Actual outcome if we comment out the `PUTC`:
  610. * `lidt`: still infinite reboot loop
  611. * `lidt2` and `lidt0`: halt apparently
  612. I think I understand that `lidt` takes as input a memory address, and the memory at that address must contain:
  613. * 2 bytes: total size of the IVT in bytes
  614. * 4 bytes: base address of the IVT. Higher byte is ignored in real mode, since addresses are not 4 bytes long.
  615. === Protected mode
  616. Hello world:
  617. ....
  618. ./run protected_mode
  619. ....
  620. Major changes from real mode:
  621. * <<vga>> must be used for output since <<bios>> is not available in protected mode.
  622. * <<protected-mode-segmentation,segmentation>> takes effect immediately, so we have to set the <<gdt>> up
  623. * we have to encode instructions differently, thus a `.code32` is needed. 16-bit mode 32-bit instructions are encodable with a special prefix.
  624. Bibliography:
  625. * http://stackoverflow.com/questions/28645439/how-do-i-enter-32-bit-protected-mode-in-nasm-assembly Initially adapted from this.
  626. * http://wiki.osdev.org/Journey_To_The_Protected_Land
  627. * http://wiki.osdev.org/Protected_Mode
  628. * https://github.com/chrisdew/xv6/blob/master/bootasm.S
  629. * https://thiscouldbebetter.wordpress.com/2011/03/17/entering-protected-mode-from-assembly/ FASM based. Did not word on first try, but looks real clean.
  630. * http://skelix.net/skelixos/tutorial02_en.html
  631. * Linux kernel v4.12 `arch/x86/include/asm/segment.h`
  632. ==== Intel protected mode example
  633. link:intel-protected/[]
  634. The <<intel-manual>> Volume 3 - 9.10 "INITIALIZATION AND MODE SWITCHING EXAMPLE" does contain an official example of how to go into protected mode.
  635. However:
  636. * the code is inside the PDF, which breaks all the formatting, so we have copied it here to this repo
  637. * TODO there is no known tool that can actually compile that syntax... although MASM should be close:
  638. ** http://computer-programming-forum.com/46-asm/6d9e8b7acea2d4cc.htm
  639. ** http://coding.derkeiler.com/Archive/Assembler/alt.lang.asm/2005-12/msg00028.html
  640. ** https://groups.google.com/forum/#!topic/comp.lang.asm.x86/9UZPQWwv-mQ 1994 comp.lang.asm.x86 topic
  641. How can those guys be in business? >:-)
  642. ==== Protected mode draw pixel
  643. TODO do it.
  644. Things get much more involved than in real mode: http://stackoverflow.com/questions/14419088/how-to-draw-a-pixel-on-the-screen-in-protected-mode-in-x86-assembly
  645. ==== Protected mode segmentation
  646. TODO: get working:
  647. ....
  648. ./run segmentation
  649. ....
  650. Expected outcome:
  651. ....
  652. x
  653. a
  654. b
  655. ....
  656. Actual outcome:
  657. ....
  658. x
  659. a
  660. ....
  661. Example of the effect on a memory access of changing the segment base address.
  662. Without segment manipulation, the output would be just:
  663. TODO: cleanup and move into main README after we get the example working: link:segmentation.adoc[]
  664. ===== Segmentation introduction
  665. First read the paging tutorial, and in particular: http://www.cirosantilli.com/x86-paging/#segmentation to get a feel for the type of register and data structure manipulation required to configure the CPU, and how segmentation compares to paging.
  666. Segmentation modifies every memory access of a given segment by:
  667. * adding an offset to it
  668. * limiting how big the segment is
  669. If an access is made at an offset larger than allowed an exception happens, which is like an interrupt, and gets handled by a previously registered handler.
  670. Segmentation could be used to implement virtual memory by assigning one segment per program:
  671. ....
  672. +-----------+--------+--------------------------+
  673. | Program 1 | Unused | Program 2 |
  674. +-----------+--------+--------------------------+
  675. ^ ^ ^ ^
  676. | | | |
  677. Start1 End1 Start2 End2
  678. ....
  679. Besides address translation, the segmentation system also managed other features such as <<protection-rings>>. TODO: how are those done in 64-bit mode?
  680. In Linux 32-bit for example, only two segments are used at all times: one at ring 0 for the kernel, and one another at privilege 3 for all user processes.
  681. ===== Segment selector
  682. In protected mode, the segment registers `CS`, `DS`, `SS`, `ES`, `FS` and `GS` contain a data structure more complex than a simple address as in real mode, which contains a single number.
  683. This 2 byte data structure is called a _segment selector_:
  684. [options="header"]
  685. |===
  686. |Position (bits) |Size (bits) |Name |Description
  687. |0
  688. |2
  689. |Request Privilege Level (RPL)
  690. |Protection ring level, from 0 to 3.
  691. |2
  692. |1
  693. |Table Indicator (TI)
  694. a|
  695. * 0: global descriptor table
  696. * 1: local descriptor table
  697. |3
  698. |13
  699. |Index
  700. a|Index of the <<segment-descriptor>> to be used from the descriptor table.
  701. |===
  702. Like in real mode, this data structure is loaded on the registers with a regular `mov` mnemonic instruction.
  703. Bibliography: <<intel-manual>> Volume 3 - 3.4.5 "Segment Descriptors".
  704. ===== GDT
  705. Global descriptor table.
  706. An in-memory array of <<segment-descriptor>> data structures:
  707. The `Index` field of the <<segment-selector>> chooses which one of those segment descriptors is to be used.
  708. The base address is set with the `lgdt` instruction, which loads from memory a 6 byte structure:
  709. [options="header"]
  710. |===
  711. |Position (bytes) |Size (bytes) |Description
  712. |0
  713. |2
  714. |Number of entries in the table
  715. |2
  716. |4
  717. |Base address of the table
  718. |===
  719. Bibliography:
  720. * https://en.wikipedia.org/wiki/Global_Descriptor_Table
  721. * http://wiki.osdev.org/GDT
  722. ====== Local descriptor table
  723. TODO vs global?
  724. ====== Null segment selector
  725. <<intel-manual>> Volume 3 - 3.4.2 "Segment Selectors" says that we can't use the first entry of the GDT:
  726. ____
  727. The first entry of the GDT is not used by the processor. A segment selector that points to this entry of the GDT (that is, a segment selector with an index of 0 and the TI flag set to 0) is used as a “null segment selector.” The processor does not generate an exception when a segment register (other than the CS or SS registers) is loaded with a null selector. It does, however, generate an exception when a segment register holding a null selector is used to access memory. A null selector can be used to initialize unused segment registers. Loading the CS or SS register with a null segment selector causes a general-protection exception (#GP) to be generated.
  728. ____
  729. ===== Segment descriptor
  730. A data structure that is stored in the <<gdt>>.
  731. Clearly described on the <<intel-manual>> Volume 3 - 3.4.5 "Segment Descriptors" and in particular Figure 3-8 "Segment Descriptor".
  732. The Linux kernel v4.2 encodes it at: `arch/x86/include/asm/desc_defs.h` in `struct desc_struct`
  733. ===== Protection rings
  734. https://stackoverflow.com/questions/18717016/what-are-ring-0-and-ring-3-in-the-context-of-operating-systems/44483439#44483439
  735. TODO example. Jump to userspace, do something naughty, handler interrupt in kernel land.
  736. ==== IDT
  737. Interrupt descriptor table.
  738. Protected mode analogue to the <<ivt>>:
  739. ....
  740. ./run idt
  741. ....
  742. outcome:
  743. ....
  744. int 0 handled
  745. ....
  746. Handle interrupt 1 instead of 0:
  747. ....
  748. ./run idt1
  749. ....
  750. outcome:
  751. ....
  752. int 1 handled
  753. ....
  754. Print `00000020\n` at `18.2 Hz` with the <<pit>>:
  755. ....
  756. ./run pit_protected
  757. ....
  758. Bibliography:
  759. * https://wiki.osdev.org/Interrupt_Descriptor_Table
  760. * https://en.wikipedia.org/wiki/Interrupt_descriptor_table
  761. * http://www.jamesmolloy.co.uk/tutorial_html/4.-The%20GDT%20and%20IDT.html
  762. The first 32 handlers are reserved by the processor and have predefined meanings, as specified in the <<intel-manual>> Volume 3 Table 3-3. "Intel 64 and IA-32 General Exceptions".
  763. In the Linux kernel, https://github.com/torvalds/linux/blob/v4.2/arch/x86/entry/entry_64.S sets them all up: each `idtentry divide_error` call sets up a new one.
  764. ===== IDT divide by zero
  765. Handle a division by zero:
  766. ....
  767. ./run idt_zero_divide
  768. ....
  769. Outcome:
  770. ....
  771. division by zero handled
  772. ....
  773. Division by zero causes a Divide Error which Intel notes as `#DE`.
  774. It is then handled by IDT 0.
  775. DEs are not only for division by zero: they also happens on overflow. TODO example.
  776. ==== SMP
  777. http://stackoverflow.com/questions/980999/what-does-multicore-assembly-language-look-like/33651438#33651438
  778. Symmetric multiprocessing https://en.wikipedia.org/wiki/Symmetric_multiprocessing
  779. ....
  780. ./run smp
  781. ....
  782. Outcome:
  783. ....
  784. SMP started
  785. ....
  786. Implies that SMP worked because a spinlock was unlocked by the second processor.
  787. Try commenting out waking up the second processor and see it not get printed.
  788. ==== Paging
  789. Verbose beginner's tutorial: http://www.cirosantilli.com/x86-paging/
  790. ....
  791. ./run paging
  792. ....
  793. Outcome:
  794. ....
  795. 00001234
  796. 00005678
  797. ....
  798. Implies that paging worked because we printed and modified the same physical address with two different virtual addresses.
  799. Requires <<protected-mode>>.
  800. ===== Page fault
  801. Generate and handle a page fault:
  802. ....
  803. ./run page_fault
  804. ....
  805. Outcome:
  806. ....
  807. Page fault handled. Error code:
  808. 00000002
  809. ....
  810. This is printed from a page fault handler that we setup an triggered by writing to an unmapped address.
  811. === IA-32e mode
  812. Wikipedia seems to call it long mode: https://en.wikipedia.org/wiki/Long_mode
  813. Contains two sub-modes: <<64-bit-mode>> and <<compatibility-mode>>.
  814. This controlled by the `CS.L` bit of the segment descriptor.
  815. It appears that it is possible for user programs to modify that during execution from userland: http://stackoverflow.com/questions/12716419/can-you-enter-x64-32-bit-long-compatibility-sub-mode-outside-of-kernel-mode
  816. TODO vs <<protected-mode>>.
  817. === 64-bit mode
  818. 64-bit is the major mode of operation, and enables the full 64 bit instructions.
  819. === Compatibility mode
  820. Compatibility mode emulates IA-32 and allows to run 32 and 16 bit code.
  821. But 64 bit Linux and Windows don't seem to allow 16 bit code anymore?
  822. * http://stackoverflow.com/questions/27868394/switch-from-64-bit-long-mode-to-32-bit-compatibility-mode-on-x64
  823. * https://stackoverflow.com/questions/7829058/how-to-run-16-bit-code-on-32-bit-linux
  824. * https://superuser.com/questions/140953/why-cant-a-64-bit-os-run-a-16-bit-application
  825. Compatibility vs protected: https://stackoverflow.com/questions/20848412/modes-of-intel-64-cpu
  826. == in and out instructions
  827. x86 has dedicated instructions for certain IO operations: `in` and `out`.
  828. These instructions take an IO address which identifies which hardware they will communicate to.
  829. The IO ports don't seem to be standardized, like everything else: http://stackoverflow.com/questions/14194798/is-there-a-specification-of-x86-i-o-port-assignment
  830. The Linux kernel wraps those instructions with the `inb` and `outb` family of instructions:
  831. ....
  832. man inb
  833. man outb
  834. ....
  835. === Memory mapped vs port mapped IO
  836. Not all instruction sets have dedicated instructions such as `in` and `out` for IO.
  837. In ARM for example, everything is done by writing to magic memory addresses.
  838. The dedicated `in` and `out` approach is called "port mapped IO", and the approach of the magic addresses "memory mapp"
  839. From an interface point of view, I feel that memory mapped is more elegant: port IO simply creates a second addresses space.
  840. TODO: are there performance considerations when designing CPUs?
  841. http://superuser.com/questions/703695/difference-between-port-mapped-and-memory-mapped-access
  842. === PS/2 keyboard
  843. ....
  844. ./run ps2_keyboard
  845. ....
  846. Whenever you press a key down or up, the keyboard hex scancode is printed to the screen.
  847. Uses the PS/2 keyboard controller on `in 60h`: http://wiki.osdev.org/%228042%22_PS/2_Controller
  848. The `in` always returns immediately with the last keyboard keycode: we then just poll for changes and print only the changes.
  849. Scancode tables: TODO: official specs?
  850. * https://en.wikipedia.org/wiki/Scancode#PC_compatibles
  851. * http://flint.cs.yale.edu/cs422/doc/art-of-asm/pdf/APNDXC.PDF
  852. TODO do this with the interrupt table instead of `in`.
  853. === PS/2 mouse
  854. TODO create an example:
  855. * http://wiki.osdev.org/Mouse_Input
  856. * Random threads with source code, ah those OS devs:
  857. ** https://forum.osdev.org/viewtopic.php?t=10247
  858. ** https://forum.osdev.org/viewtopic.php?t=24277
  859. * https://courses.engr.illinois.edu/ece390/books/labmanual/io-devices-mouse.html
  860. I am so going to make a pixel drawing program with this.
  861. === RTC
  862. Real Time Clock:
  863. * http://wiki.osdev.org/RTC
  864. * http://wiki.osdev.org/CMOS
  865. * https://en.wikipedia.org/wiki/Real-time_clock
  866. ....
  867. ./run rtc
  868. ....
  869. Gives wall date and time with precision of seconds, every second, e.g.:
  870. ....
  871. 00 01 02 03 04 10
  872. ....
  873. means:
  874. ____
  875. 3rd April 2010, 02 hours 01 minute and 00 seconds.
  876. ____
  877. Uses `out 70h` and `in 71h` to query the hardware.
  878. This hardware must therefore use a separate battery to keep going when we turn off the computer or remove the laptop battery.
  879. We can control the initial value in QEMU with the option:
  880. ....
  881. -rtc base='2010-04-03T02:01:00'
  882. ....
  883. The RTC cannot give accuracy greater than seconds. For that, consider the <<pit>>, or the <<hpet>>.
  884. Bibliography:
  885. * http://stackoverflow.com/questions/1465927/how-can-i-access-system-time-using-nasm
  886. * https://github.com/torvalds/linux/blob/v4.2/arch/x86/kernel/rtc.c#L121
  887. === PIT
  888. Programmable Interval Timer:
  889. * https://en.wikipedia.org/wiki/Programmable_interval_timer
  890. * http://wiki.osdev.org/PIT
  891. * https//en.wikipedia.org/wiki/Intel_8253 that is the circuit ID for the PIT.
  892. * http://kernelx.weebly.com/programmable-interval-timer.html
  893. Superseded by the <<hpet>>.
  894. Print `a\n` with the minimal frequency possible of `0x1234DD / 0xFFFF = 18.2 Hz`:
  895. ....
  896. ./run pit
  897. ....
  898. Make the PIT generate a single interrupt instead of a frequency:
  899. ....
  900. ./run pit_once
  901. ....
  902. Outcome:
  903. ....
  904. a
  905. ....
  906. TODO I think this counts down from the value value in channel 0, and therefore allows to schedule a single event in the future.
  907. The PIT can generate periodic interrupts (or <<pc-speaker,sound>>!) with a given frequency to `IRQ0`, which on real mode maps to interrupt 8 by default.
  908. Major application: interrupt the running process to allow the OS to schedule processes.
  909. The PIT 3 channels that can generate 3 independent signals
  910. * channel 0 at port `40h`: generates interrupts
  911. * channel 1 at port `41h`: not to be used for some reason
  912. * channel 2 at port `42h`: linked to the speaker to generate sounds
  913. Port `43h` is used to control signal properties except frequency, which goes in the channel ports, for the 3 channels.
  914. === PIT frequency
  915. We don't control the frequency of the PIT directly, which is fixed at `0x1234DD`.
  916. Instead, we control a frequency divisor. This is a classic type of discrete electronic circuit: https://en.wikipedia.org/wiki/Frequency_divider
  917. The magic frequency comes from historical reasons to reuse television hardware according to link:https://wiki.osdev.org/Programmable_Interval_Timer[], which in turn is likely influenced by some physical properties of crystal oscillators.
  918. The constant `1193181 == 0x1234DD` has 2 occurrences on Linux 4.16.
  919. ==== HPET
  920. Newer <<pit>>.
  921. TODO example.
  922. * https://en.wikipedia.org/wiki/High_Precision_Event_Timer
  923. * https://wiki.osdev.org/HPET
  924. ==== PC speaker
  925. http://wiki.osdev.org/PC_Speaker
  926. ....
  927. ./run pc_speaker
  928. ....
  929. Outcome: produces a foul noisy noise using the PC speaker hardware on `out 61h`
  930. QEMU requires the option:
  931. ....
  932. -soundhw pcspk
  933. ....
  934. The beep just uses the <<pit>> Channel 2 to generate the frequency.
  935. Extracted from: https://github.com/torvalds/linux/blob/v4.2/arch/x86/realmode/rm/wakemain.c#L38 The kernel has a Morse code encoder using it!
  936. Bibliography:
  937. * https://courses.engr.illinois.edu/ece390/books/labmanual/io-devices-speaker.html
  938. * http://fly.srk.fer.hr/GDM/articles/sndmus/speaker1.html
  939. == Video mode
  940. There are several video modes.
  941. Modes determine what interrupt functions can be used.
  942. There are 2 main types of modes:
  943. * text, where we operate character-wise
  944. * video, operate byte-wise
  945. Modes can be set with `int 0x10` and `AH = 0x00`, and get with `AH = 0x0F`
  946. The most common modes seem to be:
  947. * 0x01: 40x25 Text, 16 colors, 8 pages
  948. * 0x03: 80x25 Text, 16 colors, 8 pages
  949. * 0x13: 320x200 Graphics, 256 colors, 1 page
  950. You can add 128 to the modes to prevent them from clearing the screen.
  951. Taken from: https://courses.engr.illinois.edu/ece390/books/labmanual/graphics-int10h.html
  952. A larger list: http://www.columbia.edu/~em36/wpdos/videomodes.txt
  953. See also: http://wiki.osdev.org/How_do_I_set_a_graphics_mode
  954. === Video mode 13h
  955. https://en.wikipedia.org/wiki/Mode_13h
  956. Example at: <<bios-draw-pixel>>
  957. Video Mode `13h` has: 320 x 200 Graphics, 256 colors, 1 page.
  958. The color encoding is just an arbitrary palette that fits 1 byte, it is not split colors like R R R G G G B B or anything mentioned at: https://en.wikipedia.org/wiki/8-bit_color. Related: http://stackoverflow.com/questions/14233437/convert-normal-256-color-to-mode-13h-version-color
  959. === VGA
  960. * https://en.wikipedia.org/wiki/Video_Graphics_Array
  961. * https://en.wikipedia.org/wiki/VGA-compatible_text_mode
  962. TODO: what is it exactly?
  963. BIOS cannot be used when we move into <<protected-mode>>, but we can use the VGA interface to get output out of our programs.
  964. Have a look at the macros prefixed with `VGA_` inside link:common.h[].
  965. == Power
  966. === Shutdown
  967. http://wiki.osdev.org/Shutdown
  968. === Reboot
  969. http://stackoverflow.com/questions/32682152/how-to-reboot-in-x86-assembly-from-16-bit-real-mode
  970. Infinite reboot loop on emulator!
  971. ....
  972. ./run reboot
  973. ....
  974. TODO why does it work?
  975. === APM
  976. Turn on and immediately shutdown the system closing QEMU:
  977. ....
  978. ./run apm_shutdown
  979. ....
  980. Fancier version copied from http://wiki.osdev.org/APM (TODO why is that better):
  981. ....
  982. ./run apm_shutdown2
  983. ....
  984. https://en.wikipedia.org/wiki/Advanced_Power_Management
  985. http://wiki.osdev.org/APM
  986. Older than <<acpi>> and simpler.
  987. By Microsoft in 1995. Spec seems to be in RTF format...
  988. Can't find the URL. A Google cache: https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&ved=0CB0QFjAAahUKEwj7qpLN_4XIAhWCVxoKHa_nAxY&url=http%3A%2F%2Fdownload.microsoft.com%2Fdownload%2F1%2F6%2F1%2F161ba512-40e2-4cc9-843a-923143f3456c%2FAPMV12.rtf&usg=AFQjCNHoCx8gHv-w08Dn_Aoy6Q3K3DLWRg&sig2=D_66xvI7Y2n1cvyB8d2Mmg
  989. Bibliography:
  990. * http://wiki.osdev.org/Shutdown
  991. * http://stackoverflow.com/questions/21463908/x86-instructions-to-power-off-computer-in-real-mode
  992. * http://stackoverflow.com/questions/678458/shutdown-the-computer-using-assembly
  993. * http://stackoverflow.com/questions/3145569/how-to-power-down-the-computer-from-a-freestanding-environment
  994. === ACPI
  995. TODO example
  996. ACPI https://en.wikipedia.org/wiki/Advanced_Configuration_and_Power_Interface
  997. Newer and better.
  998. Now managed by the same group that manages UEFI.
  999. Spec:
  1000. * current: http://uefi.org/specifications
  1001. * old: http://www.uefi.org/acpi/specs
  1002. == UEFI
  1003. https://en.wikipedia.org/wiki/Unified_Extensible_Firmware_Interface
  1004. Successor for <<bios>>.
  1005. Made by Intel, mostly MIT open source, which likely implies that vendors will hack away closed source versions.
  1006. link:https://mjg59.dreamwidth.org/10014.html[Matthew Garrett says] it is huge: larger than Linux without drivers.
  1007. Since it is huge, it inevitably contains bugs. Garret says that Intel sometimes does not feel like updating the firmware with bugfixes.
  1008. UEFI offers a large API comparable to what most people would call an operating system:
  1009. * https://software.intel.com/en-us/articles/uefi-application mentions a POSIX C library port
  1010. * https://lwn.net/Articles/641244/ mentions a Python interpreter port!
  1011. ARM is considering an implementation https://wiki.linaro.org/ARM/UEFI
  1012. === UEFI example
  1013. ....
  1014. make -C uefi run
  1015. ....
  1016. TODO get a hello world program working:
  1017. * http://www.rodsbooks.com/efi-programming/hello.html Best source so far: allowed me to compile the hello world! TODO: how to run it now on QEMU and real hardware?
  1018. * https://fedoraproject.org/wiki/Using_UEFI_with_QEMU
  1019. * https://wiki.ubuntu.com/UEFI/OVMF
  1020. * https://github.com/tqh/efi-example
  1021. Running without image gives the UEFI shell, and a Linux kernel image booted fine with it: link:http://unix.stackexchange.com/a/228053/32558[], so we just need to generate the image.
  1022. The blob `uefi/ovmf.fd` IA32 r15214 was downloaded from: https://sourceforge.net/projects/edk2/files/OVMF/OVMF-IA32-r15214.zip/download TODO: automate building it from source instead, get rid of the blob, and force push it away from history. It seems that they have moved to GitHub at last: https://github.com/tianocore/tianocore.github.io/wiki/How-to-build-OVMF/e372aa54750838a7165b08bb02b105148e2c4190
  1023. === UEFI Bibliography
  1024. * https://www.youtube.com/watch?v=V2aq5M3Q76U hardcore kernel dev Matthew Garrett saying how bad UEFI is
  1025. * https://wiki.archlinux.org/index.php/Unified_Extensible_Firmware_Interface
  1026. * http://wiki.osdev.org/UEFI
  1027. == Coreboot
  1028. TODO minimal examples.
  1029. https://en.wikipedia.org/wiki/Coreboot
  1030. https://www.coreboot.org
  1031. Open source hippie freedom loving cross platform firmware that attempts to replace BIOS and UEFI for the greater good of mankind.
  1032. == GRUB
  1033. link:grub/[] TODO cleanup and exemplify everything in that file. Some hosty stuff needs to go out maybe.
  1034. === GRUB chainloader
  1035. ....
  1036. make -C grub/chainloader run
  1037. ....
  1038. Outcome: you are left in an interactive GRUB menu with two choices:
  1039. * `hello-world`: go into a hello world OS
  1040. * `self +1`: reload ourselves, and almost immediately reload GRUB and fall on the same menu as before
  1041. This example illustrates the `chainloader` GRUB command, which just loads a boot sector and runs it: https://www.gnu.org/software/grub/manual/grub/html_node/chainloader.html
  1042. This is what you need to boot systems like Windows which GRUB does not know anything about: just point to their partition and let them do the job.
  1043. Both of the menu options are implemented with `chainloader`:
  1044. * `hello-world`:
  1045. +
  1046. Loads a given image file within the partition.
  1047. +
  1048. After build, `grub-mkrescue` creates a few filesystems, and `grub/chainloader/iso/boot/main.img` is placed inside one of those filesystems.
  1049. +
  1050. This illustrates GRUB's awesome ability to understand certain filesystem formats, and fetch files from them, thus allowing us to pick between multiple operating systems with a single filesystem.
  1051. +
  1052. It is educational to open up the generated `grub/chainloader/main.img` with the techniques described at https://askubuntu.com/questions/69363/mount-single-partition-from-image-of-entire-disk-device/673257#673257 to observe that the third partition of the image is a VFAT filesystem, and that it contains the `boot/main.img` image as a regular file.
  1053. * `self +1`: uses the syntax:
  1054. +
  1055. ....
  1056. chainloader +1
  1057. ....
  1058. +
  1059. which reloads the first sector of the current partition, and therefor ourselves.
  1060. TODO: why does it fail for hybrid ISO images? http://superuser.com/questions/154134/grub-how-to-boot-into-iso-partition#comment1337357_154271
  1061. === GRUB linux
  1062. TODO get working.
  1063. OK, let's have some fun and do the real thing!
  1064. ....
  1065. make -C grub/linux run
  1066. ....
  1067. Outcome: GRUB menu with a single `Buildroot` entry. When you select it, a tiny pre-built Linux image boots from: https://github.com/cirosantilli/linux-kernel-module-cheat
  1068. Actual outcome: after selecting the entry, nothing shows on the screen. Even if we fix this, we will then also need to provide a rootfs somehow: the `initrd` GRUB command would be a simple method, that repo can also generate initrd images: https://github.com/cirosantilli/linux-kernel-module-cheat/tree/c06476bfc821659a4731d49e808f45e8c509c5e1#initrd Maybe have look under Buildroot `boot/grub2` and copy what they are doing there.
  1069. The GRUB command is of form:
  1070. ....
  1071. linux /boot/bzImage root=/dev/sda1 console=tty1
  1072. ....
  1073. so we see that the kernel boot parameters are passed right there, for example try to change the value of the `printk.time` parameter:
  1074. ....
  1075. printk.time=y
  1076. ....
  1077. and see how the dmesg times not get printed anymore.
  1078. == Multiboot
  1079. https://en.wikipedia.org/wiki/Multiboot_Specification
  1080. Standard created by GRUB for booting OSes.
  1081. Multiboot files are an extension of ELF files with a special header.
  1082. Advantages: GRUB does housekeeping magic for you:
  1083. * you can store the OS as a regular file inside a filesystem
  1084. * your program starts in 32-bit mode already, not 16 bit real mode
  1085. * it gets the available memory ranges for you
  1086. Disadvantages:
  1087. * more boilerplate
  1088. GRUB leaves the application into a well defined starting state.
  1089. It seems that Linux does not implement Multiboot natively, but GRUB supports it as an exception: http://stackoverflow.com/questions/17909429/booting-a-non-multiboot-kernel-with-grub2
  1090. === Multiboot hello world
  1091. QEMU supports multiboot natively https://stackoverflow.com/questions/25469396/how-to-use-qemu-properly-with-multi-boot-headers/32550281#32550281:
  1092. ....
  1093. make -C multiboot/hello-world run
  1094. ....
  1095. which actually runs:
  1096. ....
  1097. qemu-system-i386 -kernel 'main.elf'
  1098. ....
  1099. where `main.elf` is the multiboot file we generated.
  1100. Outcome:
  1101. ....
  1102. hello world
  1103. ....
  1104. Or you can use `grub-mkrescue` to make a multiboot file into a bootable ISO or disk:
  1105. ....
  1106. qemu-system-x86_64 -drive file=main.img,format=raw
  1107. ....
  1108. The `main.img` file can also be burned to a USB and run on real hardware.
  1109. Example originally minimized from https://github.com/programble/bare-metal-tetris
  1110. This example illustrates the `multiboot` GRUB command: https://www.gnu.org/software/grub/manual/grub/html_node/multiboot.html
  1111. === osdev multiboot hello world
  1112. We also track here the code from: link:http://wiki.osdev.org/Bare_Bones[]:
  1113. ....
  1114. make -C multiboot/osdev run
  1115. ....
  1116. Outcome:
  1117. ....
  1118. hello world
  1119. ....
  1120. This is interesting as it uses C as much as possible with some GAS where needed.
  1121. This should serve as a decent basis for starting a pet OS. But please don't, there are enough out there already :-)
  1122. == Tests
  1123. === Unit tests
  1124. Tests for utilities defined in this repo, as opposed to x86 or external firmware concepts.
  1125. TODO: implement: link:test_vga_print_bytes.S[]
  1126. ==== PRINT_BYTES
  1127. ....
  1128. ./run test_print_bytes
  1129. ....
  1130. Outcome:
  1131. ....
  1132. 40 41 42 43 44 45 46 47
  1133. 48 49 4A 4B 4C 4D 4E 4F
  1134. 50
  1135. ....
  1136. ==== PIT_SLEEP_TICKS
  1137. ....
  1138. ./run test_pit_sleep_ticks
  1139. ....
  1140. Outcome: print `a\n` with frequency 2Hz.
  1141. === Test hardware
  1142. ==== ThinkPad T400
  1143. Most of this repo was originally tested on a link:https://www.cnet.com/products/lenovo-thinkpad-t400/specs/[ThinkPad T400].
  1144. Unfortunately it broke and I threw it away, and I didn't write down the exact specs before doing so, notably the bootloader version.
  1145. == About
  1146. === System vs userland
  1147. This repository covers only things that can only be done from ring 0 (system) and not ring 3 (userland).
  1148. Ring 3 is covered at: https://github.com/cirosantilli/x86-assembly-cheat
  1149. An overview of rings 0 and 3 can be found at: https://stackoverflow.com/questions/18717016/what-are-ring-0-and-ring-3-in-the-context-of-operating-systems/44483439#44483439
  1150. === One minimal concept per OS
  1151. There are a few tutorials that explain how to make an operating system and give examples of increasing complexity with more and more functionality added.
  1152. This is not one of them.
  1153. The goal of this repository is to use the minimal setup possible to be able to observe _a single_ low-level programming concept for each minimal operating system we create.
  1154. This is not meant provide a template from which you can write a real OS, but instead to illustrate how those low-level concepts work in isolation, so that you can use that knowledge to implement operating systems or drivers.
  1155. Minimal examples are useful because it is easier to observe the requirements for a given concept to be observable.
  1156. Another advantage is that it is easier to DRY up minimal examples (here done simply through `#include` and macros), which is much harder on progressive OS template tutorials, which tend to repeat big chunks of code between the examples.
  1157. === To C or not to C
  1158. Using C or not is a hard choice.
  1159. It does make it much easier to express higher level ideas, and gives portability.
  1160. But in the end, it increases the complexity that one has to understand, so we've stayed away from it.
  1161. === NASM
  1162. ....
  1163. cd nasm/
  1164. ./run bios_hello_world
  1165. ....
  1166. While NASM is a bit more convenient than GAS to write a boot sector, I think it is just not worth it.
  1167. When writing an OS in C, we are going to use GCC, which already uses GAS. So it's better to reduce the number of assemblers to one and stick to GAS only.
  1168. Right now, this directory is not very DRY since NASM is secondary to me, so it contains mostly some copy / paste examples.
  1169. On top of that, GAS also supports other architectures besides x86, so learning it is more useful in that sense.
  1170. === Macros vs functions
  1171. Using macros for now on link:common.h[] instead of functions because it simplifies the linker script.
  1172. But the downsides are severe:
  1173. * no symbols to help debugging. TODO: I think there are assembly constructs for that.
  1174. * impossible to step over method calls: you have to step into everything. TODO: `until`?
  1175. * larger output, supposing I can get linker gc for unused functions working, see `--gc-section`, which is for now uncertain.
  1176. +
  1177. If I can get this working, I'll definitely move to function calls.
  1178. +
  1179. The problem is that if I don't, every image will need a stage 2 loader. That is not too serious though, it could be added to the `BEGIN`.
  1180. +
  1181. It seems that `ld` can only remove sections, not individual symbols: http://stackoverflow.com/questions/6687630/c-c-gcc-ld-remove-unused-symbols With GCC we can use `-ffunction-sections -fdata-sections` to quickly generate a ton of sections, but I don't thing GAS supports that...
  1182. ==== Macro conventions
  1183. Every "function-like macro" in link:common.h[] must maintain the state of general purpose registers.
  1184. Flags are currently not maintained.
  1185. `%sp` cannot be used to pass most arguments.
  1186. We don't care about setting `%bp` properly at the moment.
  1187. === Linux is open source
  1188. Always try looking into the Linux kernel to find how those CPU capabilities are used in a "real" OS.
  1189. === Pre-requisites
  1190. OS dev is one of the most insanely hard programming tasks a person can undertake, and will push your knowledge of several domains to the limit.
  1191. Knowing the following will help a lot:
  1192. * userland x86 assembly: https://github.com/cirosantilli/assembly-cheat
  1193. * compilation, linking and ELF format basics
  1194. * GDB debugging
  1195. While it is possible to learn those topics as you go along, and it is almost certain that you will end up learning more about them, we will not explain them here in detail.
  1196. == Bibliography
  1197. === Intel manual
  1198. We are interested mostly in the "Intel Manual Volume 3 System Programming Guide", where system programming basically means "OS stuff" or "bare metal" as opposed to userland present in the other manuals.
  1199. This repository quotes by default the following revision: 325384-056US September 2015 https://web.archive.org/web/20151025081259/http://www.intel.com/content/dam/www/public/us/en/documents/manuals/64-ia-32-architectures-software-developer-system-programming-manual-325384.pdf
  1200. === Small educational projects
  1201. Fun, educational and useless:
  1202. * https://github.com/programble/bare-metal-tetris tested on Ubuntu 14.04. Just works.
  1203. +
  1204. Has Multiboot and El Torito. Uses custom linker script.
  1205. +
  1206. Almost entirely in C `-nostdlib`, with very few inline `asm` commands, and a small assembly entry point. So a good tutorial in how to do the bridge.
  1207. * https://github.com/daniel-e/tetros Tetris that fits into bootloader.
  1208. * https://github.com/arjun024/mkeykernel, https://github.com/arjun024/mkernel
  1209. +
  1210. Worked, but bad build system: not `Makefile` or `.gitignore`.
  1211. * https://github.com/Overv/MineAssemble
  1212. The following did not work on my machine out of the box:
  1213. * https://github.com/apparentlymart/ToyOS
  1214. * https://github.com/rde1024/toyos
  1215. === Tutorials
  1216. * https://farid.hajji.org/en/blog/46-hello-world-on-the-bare-metal
  1217. * https://arobenko.gitbooks.io/bare_metal_cpp/content/
  1218. ==== Educational NIXes
  1219. One complexity order above the minimal tutorials, one below actual kernels
  1220. * http://www.xinu.cs.purdue.edu/
  1221. * https://pdos.csail.mit.edu/6.828/2014/xv6.html
  1222. * https://en.wikipedia.org/wiki/MINIX, influenced Linux
  1223. ==== Educational non-NIXes
  1224. * https://github.com/intermezzOS/book
  1225. * https://github.com/flosse/rust-os-comparison
  1226. === Multi collaborator websites
  1227. * osdev.org is a major source for this.
  1228. ** http://wiki.osdev.org/C%2B%2B_Bare_Bones
  1229. ** http://wiki.osdev.org/Text_UI
  1230. ** http://wiki.osdev.org/GUI
  1231. * http://www.osdever.net/
  1232. * https://courses.engr.illinois.edu/ece390/books/labmanual/index.html Illinois course from 2004
  1233. === Progressive tutorials
  1234. * http://www.jamesmolloy.co.uk/tutorial_html/index.html
  1235. +
  1236. Highly recommended.
  1237. +
  1238. Multiboot based kernels of increasing complexity, one example builds on the last one. Non DRY as a result.
  1239. +
  1240. Cleaned up source code: https://github.com/cirosantilli/jamesmolloy-kernel-development-tutorials
  1241. +
  1242. Well known bugs: http://wiki.osdev.org/James_Molloy's_Tutorial_Known_Bugs That's what happens when you don't use GitHub.
  1243. +
  1244. Good tutorials, author seems to master the subject.
  1245. +
  1246. But he could learn more about version control and build automation: source code inside ugly tar.gz with output files.
  1247. * https://sourceforge.net/p/oszur11/code/ci/master/tree/
  1248. +
  1249. GitHub mirror: https://github.com/cirosantilli/oszur11-operating-system-examples
  1250. +
  1251. Several examples of increasing complexity. Found at: http://stackoverflow.com/questions/7130726/writing-a-hello-world-kernel
  1252. +
  1253. Just works, but examples are non-minimal, lots of code duplication and blobs. There must be around 20 El Torito blobs in that repo.
  1254. +
  1255. Multiboot based.
  1256. * https://github.com/SamyPesse/How-to-Make-a-Computer-Operating-System
  1257. * http://www.brokenthorn.com/Resources/OSDevIndex.html
  1258. * http://skelix.net/skelixos/index_en.html
  1259. +
  1260. Cleaned up version: https://github.com/cirosantilli/skelix-os
  1261. +
  1262. Not tested yet.
  1263. +
  1264. GAS based, no multiboot used.
  1265. * https://github.com/littleosbook/littleosbook
  1266. === Actually useful
  1267. These are not meant as learning resources but rather as useful programs:
  1268. * https://github.com/scanlime/metalkit A more automated / general bare metal compilation system. Untested, but looks promising.
  1269. * Python without an "OS": https://us.pycon.org/2015/schedule/presentation/378/
  1270. === Other archs
  1271. For when we decide to port this tutorial:
  1272. ARM:
  1273. * https://github.com/bravegnu/gnu-eprog
  1274. Raspberry PI:
  1275. * https://github.com/dwelch67/raspberrypi
  1276. * https://github.com/BrianSidebotham/arm-tutorial-rpi
  1277. == LICENSE
  1278. Copyright Ciro Santilli http://www.cirosantilli.com/
  1279. https://www.gnu.org/licenses/gpl-3.0.txt[GPL v3] for executable computer program usage.
  1280. https://creativecommons.org/licenses/by-sa/4.0/[CC BY-SA v4] for human consumption usage in learning material, e.g. `.md` files, source code comments, using source code excerpts in tutorials. Recommended attribution:
  1281. * Single file adaptations:
  1282. +
  1283. ....
  1284. Based on https://github.com/cirosantilli/x86-bare-metal-examples/blob/<commit-id>/path/to/file.md under CC BY-SA v4
  1285. ....
  1286. * Multi-file adaptations:
  1287. +
  1288. ....
  1289. Based on https://github.com/cirosantilli/x86-bare-metal-examples/tree/<commit-id> under CC BY-SA v4
  1290. ....
  1291. If you want to use this work under a different license, contact the copyright owner, and he might make a good price.