Krock/dawn

Context

The devs are currently A/B testing a new method to update the game more efficiently by using chunks. This is noticeably more complex than the conventional method, hence this issue.

Motivation

The current install script is far from perfect but does serve its purpose for those who would like to use it. If this chunk-based method becomes mandatory, other 3rd party installers/launchers could benefit from any open-source implementation that is available.

API documentation (WIP)

Tree view of the data structure:

json
- Similar to the known resource
manifest file
- File names and patch offsets (?)
chunk data
- Also zstd-compressed
- Unknown relation with the files generated by the launcher

Main JSON

see updater/sophon/updater.py

Manifest file

see updater/sophon/updater.py
(old) Analysis of the Protobuf data: (in bytes, starting from offset 0)

Size      Description
4         ?
1         file 1: name length -> "fnlen"
fnlen     file 1: name (relative to the game root)
1         ? 0x12
1         ?
1         ? 0x0A
1         file 1 chunk 1: meta length -> "cmlen1", generally = 0x31
cmlen1    file 1 chunk 1: meta name
             Format: "<hash of length 17>_<hash of length 32>"
1         ? 0x12
1         file 1 chunk 1: hash length -> "chlen1", generally = 0x20
chlen1    file 1 chunk 1: hash (?)
1         ? 0x18 or 0x20
21        ?

1         file 1 chunk 2: meta length
    ...

?         file 2: name length

EDIT 2024-04-25: Received a proto definition and further explanation. Currently clarifying licensing matters.

EDIT 2024-04-26: First script prototype uploaded

Chunk data (segments?)

Source: Main JSON. URL:
- ".data .manifests[] .chunk_download .url_prefix" + "/" + <hash of length 17>_<hash of length 32>
File format: zstd-compressed
- zstd -d FILENAME -o output

Chunk data (launcher)

Source: unknown. Possibly assembled or renamed from the section above
Destination: GAMEDIR/chunk/
File name: (32 bytes), hash
- _tmp is used during download.

If you have any relevant information or hints - feel free to leave a comment below or send me an email.

## Context The devs are currently A/B testing a new method to update the game more efficiently by using chunks. This is noticeably more complex than the conventional method, hence this issue. ## Motivation The current install script is far from perfect but does serve its purpose for those who would like to use it. If this chunk-based method becomes mandatory, other 3rd party installers/launchers could benefit from any open-source implementation that is available. ## API documentation (WIP) Tree view of the data structure: * json * Similar to the known `resource` * manifest file * File names and patch offsets (?) * chunk data * Also zstd-compressed * Unknown relation with the files generated by the launcher **Main JSON** * see `updater/sophon/updater.py` **Manifest file** * see `updater/sophon/updater.py` * (old) Analysis of the Protobuf data: (in bytes, starting from offset 0) ``` Size Description 4 ? 1 file 1: name length -> "fnlen" fnlen file 1: name (relative to the game root) 1 ? 0x12 1 ? 1 ? 0x0A 1 file 1 chunk 1: meta length -> "cmlen1", generally = 0x31 cmlen1 file 1 chunk 1: meta name Format: "<hash of length 17>_<hash of length 32>" 1 ? 0x12 1 file 1 chunk 1: hash length -> "chlen1", generally = 0x20 chlen1 file 1 chunk 1: hash (?) 1 ? 0x18 or 0x20 21 ? 1 file 1 chunk 2: meta length ... ? file 2: name length ``` **EDIT 2024-04-25:** Received a proto definition and further explanation. Currently clarifying licensing matters. **EDIT 2024-04-26:** First script prototype uploaded **Chunk data** (segments?) * Source: *Main JSON*. URL: * `".data .manifests[] .chunk_download .url_prefix"` + `"/"` + `<hash of length 17>_<hash of length 32>` * File format: zstd-compressed * `zstd -d FILENAME -o output` **Chunk data (launcher)** * Source: unknown. Possibly assembled or renamed from the section above * Destination: GAMEDIR/chunk/ * File name: (32 bytes), hash * `_tmp` is used during download. ---- If you have any relevant information or hints - feel free to leave a comment below or send me an email.

As of 4.8.0, pre-download zip archives are no longer provided by the resource.json file.

Solution 1: Use the HoYoPlay launcher

Chunk-based is used by default
zip archives are used if present inside the game directory

Solution 2: Use the freshly updated update_gi.sh script (incomplete CN support)

Same CLI syntax as usual.

EDIT 2024-07-17: Update according to latest Git commit.

I tried the predownload yesterday using the updated update_gi.sh script but it wouldn't budge

~/games/genshin-impact/…/Genshin Impact/Genshin Impact Game ∇ bash ~/code/dawn/updater/update_gi.sh predownload
--- Using download application: /usr/bin/wget -c
--- Checking for predownload versions
--- Installed version: 4.7.0

--2024-07-17 00:08:54--  https://sg-hyp-api.hoyoverse.com/hyp/hyp-connect/api/getGamePackages?game_ids[]=gopR6Cufr3&launcher_id=VYTpXlbWo8
Loaded CA certificate '/etc/ssl/certs/ca-certificates.crt'
Resolving sg-hyp-api.hoyoverse.com (sg-hyp-api.hoyoverse.com)... 18.245.31.10, 18.245.31.129, 18.245.31.103, ...
Connecting to sg-hyp-api.hoyoverse.com (sg-hyp-api.hoyoverse.com)|18.245.31.10|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [application/json]
Saving to: ‘getGamePackages?game_ids[]=gopR6Cufr3&launcher_id=VYTpXlbWo8’

getGamePackages?game_ids[]=gopR6Cufr3&l     [ <=>                                                                           ]   5.00K  --.-KB/s    in 0s

2024-07-17 00:08:55 (199 MB/s) - ‘getGamePackages?game_ids[]=gopR6Cufr3&launcher_id=VYTpXlbWo8’ saved [5118]


  *  Could not find any matching update entry

 -- exit cleanup done --

Today, it's complaining about the missing patch folder for 480.

~/games/genshin-impact/…/Genshin Impact/Genshin Impact Game ж bash ~/code/dawn/updater/update_gi.sh
--- Using download application: /usr/bin/wget -c
--- Installed version: 4.7.0

--2024-07-17 22:00:21--  https://sg-hyp-api.hoyoverse.com/hyp/hyp-connect/api/getGamePackages?game_ids[]=gopR6Cufr3&launcher_id=VYTpXlbWo8
Loaded CA certificate '/etc/ssl/certs/ca-certificates.crt'
Resolving sg-hyp-api.hoyoverse.com (sg-hyp-api.hoyoverse.com)... 3.165.136.46, 3.165.136.123, 3.165.136.16, ...
Connecting to sg-hyp-api.hoyoverse.com (sg-hyp-api.hoyoverse.com)|3.165.136.46|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [application/json]
Saving to: ‘getGamePackages?game_ids[]=gopR6Cufr3&launcher_id=VYTpXlbWo8’

getGamePackages?game_ids[]=gopR6Cufr3&l     [ <=>                                                                           ]   5.00K  --.-KB/s    in 0s

2024-07-17 22:00:22 (282 MB/s) - ‘getGamePackages?game_ids[]=gopR6Cufr3&launcher_id=VYTpXlbWo8’ saved [5118]

--- Latest version: 4.8.0

  *  No suitable patch script found. Please check the bug tracker for details about the progress.

 -- exit cleanup done --

Since there isn't anything version-specific anymore in the patch scripts (they're not doing much besides mentioning the dns blockage), I simply copied the folder to 480 after which it happily downloads the patch.

I tried the predownload yesterday using the updated `update_gi.sh` script but it wouldn't budge ``` ~/games/genshin-impact/…/Genshin Impact/Genshin Impact Game ∇ bash ~/code/dawn/updater/update_gi.sh predownload --- Using download application: /usr/bin/wget -c --- Checking for predownload versions --- Installed version: 4.7.0 --2024-07-17 00:08:54-- https://sg-hyp-api.hoyoverse.com/hyp/hyp-connect/api/getGamePackages?game_ids[]=gopR6Cufr3&launcher_id=VYTpXlbWo8 Loaded CA certificate '/etc/ssl/certs/ca-certificates.crt' Resolving sg-hyp-api.hoyoverse.com (sg-hyp-api.hoyoverse.com)... 18.245.31.10, 18.245.31.129, 18.245.31.103, ... Connecting to sg-hyp-api.hoyoverse.com (sg-hyp-api.hoyoverse.com)|18.245.31.10|:443... connected. HTTP request sent, awaiting response... 200 OK Length: unspecified [application/json] Saving to: ‘getGamePackages?game_ids[]=gopR6Cufr3&launcher_id=VYTpXlbWo8’ getGamePackages?game_ids[]=gopR6Cufr3&l [ <=> ] 5.00K --.-KB/s in 0s 2024-07-17 00:08:55 (199 MB/s) - ‘getGamePackages?game_ids[]=gopR6Cufr3&launcher_id=VYTpXlbWo8’ saved [5118] * Could not find any matching update entry -- exit cleanup done -- ``` Today, it's complaining about the missing patch folder for 480. ``` ~/games/genshin-impact/…/Genshin Impact/Genshin Impact Game ж bash ~/code/dawn/updater/update_gi.sh --- Using download application: /usr/bin/wget -c --- Installed version: 4.7.0 --2024-07-17 22:00:21-- https://sg-hyp-api.hoyoverse.com/hyp/hyp-connect/api/getGamePackages?game_ids[]=gopR6Cufr3&launcher_id=VYTpXlbWo8 Loaded CA certificate '/etc/ssl/certs/ca-certificates.crt' Resolving sg-hyp-api.hoyoverse.com (sg-hyp-api.hoyoverse.com)... 3.165.136.46, 3.165.136.123, 3.165.136.16, ... Connecting to sg-hyp-api.hoyoverse.com (sg-hyp-api.hoyoverse.com)|3.165.136.46|:443... connected. HTTP request sent, awaiting response... 200 OK Length: unspecified [application/json] Saving to: ‘getGamePackages?game_ids[]=gopR6Cufr3&launcher_id=VYTpXlbWo8’ getGamePackages?game_ids[]=gopR6Cufr3&l [ <=> ] 5.00K --.-KB/s in 0s 2024-07-17 22:00:22 (282 MB/s) - ‘getGamePackages?game_ids[]=gopR6Cufr3&launcher_id=VYTpXlbWo8’ saved [5118] --- Latest version: 4.8.0 * No suitable patch script found. Please check the bug tracker for details about the progress. -- exit cleanup done -- ``` Since there isn't anything version-specific anymore in the patch scripts (they're not doing much besides mentioning the dns blockage), I simply copied the folder to 480 after which it happily downloads the patch.

…
Extracting archive: ../_update_gi_download/audio_en-us_4.7.0_4.8.0_hdiff_iRPQrRliVLmISmPj.zip
--
Path = ../_update_gi_download/audio_en-us_4.7.0_4.8.0_hdiff_iRPQrRliVLmISmPj.zip
Type = zip
Physical Size = 453687654

Everything is Ok

Folders: 4
Files: 37
Size:       494654763
Compressed: 453687654

--- Patching file ./GenshinImpact_Data/StreamingAssets/AudioAssets/English(US)/2049.pck
--- Patching file ./GenshinImpact_Data/StreamingAssets/AudioAssets/English(US)/External0.pck
--- Patching file ./GenshinImpact_Data/StreamingAssets/AudioAssets/English(US)/External1.pck
--- Patching file ./GenshinImpact_Data/StreamingAssets/AudioAssets/English(US)/External10.pck
--- Patching file ./GenshinImpact_Data/StreamingAssets/AudioAssets/English(US)/External11.pck
…
--- Patching file ./GenshinImpact_Data/StreamingAssets/AudioAssets/English(US)/External7.pck
--- Patching file ./GenshinImpact_Data/StreamingAssets/AudioAssets/English(US)/External8.pck
--- Patching file ./GenshinImpact_Data/StreamingAssets/AudioAssets/English(US)/External9.pck
--- Removing downloaded archives ...
jq: error (at <stdin>:1): Cannot index array with string "version"
 -- exit cleanup done --
 ?=5

Must be from this code:

if [ -n "$sdk_json" ]; then
	sdk_version=`<<< "$sdk_json" jq -r -M ".version"`
	sed -i "s/^sdk_version=.*/sdk_version=${sdk_version}/" "$CONFIG_FILE"
fi

There is no sdk_version in my config.ini anyway and game_version shows 4.8.0 already.

@FichteFoll This issue is mainly about the "new" python script. Please open a new issue if you run into issues with the update_gi.sh script.

The errors should be resolved as of ~~5c2441c~~ ff5bb86.

Kirjaudu sisään osallistuaksesi tähän keskusteluun.

#474 Chunk-based updater reverse engineering

Context

Motivation

API documentation (WIP)