diff options
| -rw-r--r-- | README.md | 184 |
1 files changed, 159 insertions, 25 deletions
diff --git a/README.md b/README.md index 2a67c00d..ee787c04 100644 --- a/README.md +++ b/README.md @@ -1,47 +1,181 @@ # HSMusic -HSMusic, short for the *Homestuck Music Wiki*, is a revitalization and reimagining of [earlier][fandom] [projects][nsnd] archiving and celebrating the expansive history of Homestuck official and fan music. Roughly periodic releases of the website are released at [hsmusic.wiki][hsmusic]; all development occurs in this public Git repository, which can be accessed at [notabug.org][notabug]. +HSMusic, short for the *Homestuck Music Wiki*, is a revitalization and reimagining of [earlier][fandom] [projects][nsnd] archiving and celebrating the expansive history of Homestuck official and fan music. Roughly periodic releases of the website are released at [hsmusic.wiki][hsmusic]; all development occurs in this public Git repository, which can be accessed at [github.com][github]. -## Project Structure +## Quick Start + +Install dependencies: + +- [Node.js](https://nodejs.org/en/) - we recommend using [nvm](https://github.com/nvm-sh/nvm) to install Node and keep easy track of any versions you've got installed; development is generally tested on latest but 16.x LTS should also work +- [ImageMagick](https://imagemagick.org/) - check your package manager if it's available (e.g. apt or homebrew) or follow [installation info right from the official website](https://imagemagick.org/script/download.php) + +Make a new empty folder for storing all your HSMusic repositories, then clone 'em with git: + +``` +$ cd /path/to/my/projects/ +$ mkdir hsmusic +$ cd hsmusic +$ git clone https://github.com/hsmusic/hsmusic-wiki code +Cloning into 'code'... +$ git clone https://github.com/hsmusic/hsmusic-data data +Cloning into 'data'... +$ git clone https://nebula.ed1.club/git/hsmusic-media media +Cloning into 'media'... +``` + +Install NPM dependencies (packages) used by HSMusic: + +``` +$ cd code +$ npm install +added 413 packages, and audited 612 packages in 10s +``` + +Optionally, use `npm link` to make `hsmusic` available from the command line anywhere on your device: + +``` +$ npm link +# This doesn't work reliably on every device. If it shows +# an error about permissions (and you aren't interested in +# working out the details yourself), you can just move on. +``` + +Go back to the main directory (containing all the repos) and make an empty folder for the first and subsequent builds: + +``` +$ cd .. + +$ pwd +/path/to/my/projects/hsmusic +$ ls +code/ data/ media/ +# If you don't see the above info, you've moved to the wrong directory. +# Just do cd /path/to/my/projects/hsmusic (with whatever path you created +# the main directory in) to get back. -**Disclaimer:** most of the code here *sucks*. It's been shambled together over the course of over a year, and while we're fairly confident it's all at minimum functional, we can't guarantee the same about its understandability! Still, for the official release of [hsmusic.wiki][hsmusic], we've done our best to put together a codebase which is *somewhat* navigable. The description below summarizes it: +$ mkdir out +``` -* `src/upd8.js`: "Build" code for the site. Everything specific to generating the structure and HTML content of the website is conatined in this file. As expected, it's pretty massive, and is currently undergoing some much-belated restructuring. -* `src/static`: Static code and supporting files. Everything here is wholly client-side and referenced by the generated HTML files. -* `src/common`: Code which is depended upon by both client- and server-side code. For the most part, this is constants such as directory paths, though there are a few handy algorithms here too. -* In the not quite so far past, we used to have `data` and `media` folders too. Today, for portability and convenience in project structure, those are saved in separate repositories, and you can pass hsmusic paths to them through the `--data-path` and `--media-path` options, or the `HSMUSIC_DATA` and `HSMUSIC_MEDIA` environment variables. - * Data directory: The majority of data files belonging to the wiki are here. If you were to, say, create a fork of hsmusic for some other music archival project, you'd want to change the files here. Data files are all a custom text format designed to be easy to edit, process, and maintain; they should be self-descriptive. - * Media directory: Images and other static files referenced by generated and static content across the site. Many of the files here are cover art, and their names match the automatically generated "kebab case" identifiers for tracks and albums (or a manually overridden one). -* Same for the output root: previously it was in a `site` folder; today, use `--out-path` or `HSMUSIC_OUT`! +Then build the site: -The upd8 code process was politely introduced by 2019!us back when we were beginning the site, and it's essentially the same structure followed today. In summary: +``` +# If you used npm link: +$ hsmusic --data-path data --media-path media --out-path out -1. Locate and read data files, processing them into relatively usable JS object-style formats. (The formats themselves are hard-coded and somewhat arbitrary, and are often extended when more or different data is useful.) -2. Validate the data and show any errors that might've been caught during processing. (These aren't exhaustive test cases; they're designed to catch a majority of common errors and typos.) -3. Create symlinks for static files and generate the basic directory structure for the site. -4. Generate and write HTML files containing all content. (Rather than use external templates and a complex build system, we just use template strings in combination with [a whitespace utility][fixws] and some handy tricks for manipulating strings and JS.) +# If you didn't: +$ node code/src/upd8.js --data-path data --media-path media --out-path out +``` -The majority of the code volume is generated HTML content and supporting utility functions; while we've attempted to keep the update file more or less organized, the most reliable way to navigate is to just ctrl-F for the function definitions of whatever you intend to work on. Code order isn't super strict since everything is handled by separate function calls (which all branch off of the "main" function at the end of the file). +You should get a bunch of info eventually showing the site building! It may take a while (especially since HSMusic has a lot of data nowadays). -In the past, data, HTML, and media files were all interspersed with each other. Yea, even the generated HTML files were included as part of the repository; their diffs, part of every commit. Those were dark times indeed. +If all goes according to plan and there aren't any errors, all the site HTML should have been written to the `out` directory. Use a simple HTTP server to view it in your browser: + +``` +$ cd site + +# choose your favorite HTTP server +$ npx http-server -p 8002 +$ python3 -m http.server 8002 +$ python2 -m SimpleHTTPServer 8002 +``` + +If you don't have access to an HTTP server or lack device permissions to run one, you can also just view the generated HTML files in your browser and *most* features should still work. (Try `--append-index-html` in the `hsmusic`/`upd8.js` command to make generated links more direct.) This isn't an officially supported way to develop, so there might be bugs, but most of the site should still work. + +**If you encounter any errors along the way, or would like help getting the wiki working,** please feel welcomed to reach out through the [HSMusic Community Discord Server][discord]. We're a fairly active group there and are always happy to help! **This also applies if you don't have much experience with Git, GitHub, Node, or any of the necessary tooling, and want help getting used to them.** + +## Project Structure + +### General build process + +When you run HSMusic to build the wiki, several processes happen in succession. Any errors along the way will be reported - we hope with human-readable feedback, but [pop by the Discord][discord] if you have any questions or need help understanding errors or parts of the code. + +1. Update thumbnails in the media repo so that any new images automatically get thumbnails. +2. Locate and read data files, processing them into relatively usable JS object-style formats. +3. Validate the data and show any errors caught during processing. +4. Create symlinks for static files and generate the basic directory structure for the site. +5. Generate and write HTML files (and any supporting files) containing all content. + +### Multiple repositories + +HSMusic works using a number of repositories in tandem: + +- [`hsmusic-wiki`][github-code] (colloquially "code"): The code repository, including all behavior required to process data and content from the other repositories and turn it into an actual website. This is probably the repo you're viewing right now. + - Code is written entirely in modern JavaScript, with the actual website a static combination of HTML and CSS (with inexhaustive JavaScript for certain features). + - More details about the code repository below. +- [`hsmusic-data`][github-data]: The data repository, comprising all the data which makes a given wiki what it *is*. The repository linked here is for the [Homestuck Music Wiki][hsmusic] itself, but it may be swapped out for other data repos to build other completely different wikis. + - This repo covers albums, tracks, artists, groups, and a variety of other things which make up the content of a music wiki. + - The data repo also contains all the metadata which makes one wiki unique from another: layout info, static pages (like "About & Credits"), whether or not certain site features are enabled (like "Flashes & Games" or UI for browsing groups), and so on. + - All data is written and accessed in the YAML format, and every file follows a specific structure described within this (code) repository. See below and the `src/data` directory for details. +- [`hsmusic-media`][ed1-media]: The media repository, holding all album, track, and layout media used across the site in one place. Media and organization directly corresponds to entries in the data repository; generally the data and media repositories go together and are swapped out for another together. +- *Language repo:* The language repository, holding up-to-date strings and other localization info for HSMusic. NB: This repo isn't currently online as its structure and tooling haven't been polished or properly put together yet, but it's not required for building the site. + - Strings and language info are stored in top-level JSON files within this repository. They're based off the `src/strings-default.json` file within the code repo (and don't need to provide translations for all strings to be used for site building). + +The code repository as well as the data and media repositories are require for site building, with the language repo optionally provided to add localization support to the wiki build. + +The path to each repo may be specified respectively by the `--data-path`, `--media-path`, and `--lang-path` arguments (when building the site or using e.g. data-related CLI tools). If you find it inconvenient to type or keep track of these values, you may alternatively set environment variables `HSMUSIC_DATA`, `HSMUSIC_MEDIA`, and `HSMUSIC_LANG` to provide the same values. One convenient layout for locally organizing the HSMusic repositories is shown below: + + path/to/my/projects/ + hsmusic/ + code/ <clone of hsmusic-wiki> + data/ <clone of hsmusic-data> + media/ <clone of hsmusic-media> + out/ <empty directory> (will be overwritten) + env.sh + +The `env.sh` script shown above is a straightforward utility for loading those variables into the envronment, so you don't need to type path arguments every time: + + #!/bin/bash + base="$(realpath "$(dirname ${BASH_SOURCE[0]})")" + export HSMUSIC_DATA="$base/data/" + export HSMUSIC_MEDIA="$base/media/" + # export HSMUSIC_LANG="$base/lang/" # uncomment if present + export HSMUSIC_OUT="$base/out/" + +Then use `source env.sh` when starting work from the CLI to get access to all the convenient environment variables. (This setup is written for Bash of course, but you can use the same idea to export env variables with your own shell's syntax.) + +### Code repository source structure + +The source code for HSMusic is divided across a number of source files, loosely grouped together in a number of directories: + +- `src/` + - `data/` + - `cacheable-object.js`: Backbone of how data objects (colloquially "things") store, share, and compute their properties + - `things.js`: Descriptors for all "thing" classes used across the wiki: albums, tracks, artists, groups, etc + - `validators.js`: Convenient error-throwing utilities which help ensure properties set on things follow the right format + - `yaml.js`: Mappings from YAML documents (the format used in `hsmusic-data`) to things (actual data objects), and a full set of utilities used to actually load that data from scratch + - `page/` + - All page templates (HTML content and layout metadata) are kept in source files under this directory + - `static/` + - Purely client-side files are kept here, e.g. site CSS, icon SVGs, and client-side JS + - `util/` + - Common utilities which generally may be accessed from both Node.js or the client (web browser) + - `upd8.js`: Main entry point which controls and directs site generation from start to finish + - `gen-thumbs.js`: Standalone utility also called every time HSMusic is run (unless `--skip-thumbs` is provided) which keeps a persistent cache of media MD5s and (re)generates thumbnails for new or updated image files + - `repl.js`: Standalone utility for loading all wiki data and providing a convenient REPL to run filters and transformations on data objects right from the Node.js command line + - `listing-spec.js`: Descriptors for computations and HTML templates used for the Listings part of the site + - `misc-templates.js`: General collection of HTML patterns used across page generation + - `url-spec.js`: Index of output paths where generated HTML ends up; also controls where `<a>`, `<img>`, etc tags link + - `file-size-preloader.js`: Simple utility for calculating size of files in media directory + - `strings-default.json`: Template for localization strings and index of default (English) strings used all across the site layout ## Forking hsmusic is a relatively generic music wiki software, so you're more than encouraged to create a fork for your own archival or cataloguing purposes! You're encouraged to [drop us a link][feedback] if you do - we'd love to hear from you. -Still, at present moment, a fair bit of the wiki design is baked into the update code itself - any configuration (such as getting rid of the "flashes & games") section will have you digging into the code yourself. In the future, we'd love to make the wiki software more customizable from a forking perspective, but we haven't gotten to it yet. Let us know if this is something you're interested in - we'd love to chat about what additions or changes would be useful in making a more versatile generic music wiki software! - ## Pull Requests -As mentioned, part of the focus of the hsmusic.wiki release was to create a more modular and develop-able repository. So, on the curious chance anyone would like to contribute code to the repo, such is certainly capable now! +As mentioned, part of the focus of the hsmusic.wiki release, as well as most development since, has been to create a more modular and developer-friendly repository. So, on the curious chance anyone would like to contribute code to the repo, that's more possible now than it used to be! -Still, for larger additions, I'd encourage you to throw an email or contact ([links here][feedback]) before writing all the implementation code: besides code tips which might make your life a bit easier (questions are welcome), I'd also love to discuss feature designs and values while they're still being brainstormed! That way, I don't need to tell you there are fundamental ideas or code details I'd want rebuilt - the last thing I want is anyone putting hours into code which could have been avoided being poured down the drain! +Still, for larger additions, we encourage you to [drop the main dev an email][feedback] or, better yet, [pop by the Discord][discord] before writing all the implementation code: besides code tips which might make your life a bit easier (questions are welcome), we also love to discuss feature designs and values while they're still being brainstormed! That way, nobody has to tell you there are fundamental ideas or implementation details that should be rebuilt from the ground up - the last thing we want is anyone putting hours into code that has to be replaced by another implementation before it ever ends up part of the wiki! As ever, feedback is always welcome, and may be shared via the usual links. Thank you for checking the repository out! + [ed1-media]: https://nebula.ed1.club/git/hsmusic-media/ + [discord]: https://hsmusic.wiki/discord/ [fandom]: https://homestuck-and-mspa-music.fandom.com/wiki/Homestuck_and_MSPA_Music_Wiki - [nsnd]: https://homestuck.net/music/references.html - [hsmusic]: https://hsmusic.wiki - [notabug]: https://notabug.org/hsmusic/hsmusic - [fixws]: https://www.npmjs.com/package/fix-whitespace [feedback]: https://hsmusic.wiki/feedback/ + [github]: https://github.com/hsmusic/hsmusic-wiki + [github-code]: https://github.com/hsmusic/hsmusic-wiki + [github-data]: https://github.com/hsmusic/hsmusic-data + [hsmusic]: https://hsmusic.wiki + [nsnd]: https://homestuck.net/music/references.html |