Ben, welcome to TalkGraphics. I like a challenge.

1. There is no large preview image in any Xara design file. There is a 32kB Bitmap in the Bitmap Gallery. This is not used in any design unless you explicitly copy it in. It is in the program, not the design, or on any rendered output. XPro+ rendered page backgrounds use CSS with <span> Tags or SVG fills so anything like this is tens of bytes..
I used the A-4 1-column template with Opens Sans 10pt. I created 6 pages of text (22,000 chars) and the design file was 60kB. Each rendered HTML page was 10kB. I would use multiple paged design files and treat each page as one file equivalent. I'll cover how many pages per design file later on. The overheads of rendering obviously share the other required assets (JS, CSS, WOFF). So if you have 100k files your design files would be 1GB. If you rendered all out to HTML or PDF then another 1GB.

2. It is relative easy to link to external images. You create a Placeholder and make it Repel text under. Then you include an HTML <img> Tag with src to the external file. If you want to see the actual image in the design, you would Re-generate the Placeholder.; an image smaller in size (probably) to the original would appear and become part of the design file size. Unticked, there is no overhead. It depends if you need to see the image in the design or only as rendered. Potentially, you could just drag your image in, apply its Image Filename and , crucially, Optimise Photo. The original is preserved but the design and render are kept at 192/96dpi. External linking to achieve 300dpi is clearly going to be the best for you.

3. I would search out a web widget such as MathJax. You add its code to your design website head and put your editable equation into another Placeholder.

4. PDF/X 300dpi is more than adequate.

For books 50 - 800 pages, I would split into sections/chapters of up to 100 pages. Export as PDF/X and use a utility to merge the sections/chapters into your book. I have handled 500-page designs without images and Xara barely copes. Its newest version may, I have not checked.

If you choose to isolate each file then you loose the power of linkage.
I would chuck/gather themed content into a multi-page design file, where, for little overhead, you can hyperlink easily across pages. With a little effort, your links can cross into other rendered chunks, making a cohesive whole.

An alternative is to change your text files into MarkDown files. You get the benefit of using a plain text editor but display as HTML and print to PDF.
I would use something like GitHub where you can have your own private repository. I use similar but self-hosted with Datenstrom Yellow but your project is probably too big for that.

Acorn