mirror of
https://github.com/wekan/wekan.git
synced 2026-11-20 09:45:04 -05:00
WeKan and Trello .zip imports use the shared upload, limit and disk check; .mbox and .enex export a page at a time.
The WeKan .zip route and the Trello package route wrote their uploads with caps of their own (5 GB and 200 MB, 1 GB unpacked). They now go through the same receiver as every Import page upload: the Admin Panel's largest import (else WEKAN_IMPORT_ZIP_MAX_BYTES for the WeKan zip), or with none the free disk space, checked before and while the file is written, and a Trello package may unpack to as much as that. The .mbox and .enex exports are written to the response 500 cards at a time, waiting for the socket when it is full, instead of being built whole in memory; the pages together are the same file. TODO.md keeps what is left: streamed reading of large JSON and text, and the other formats' exports. Thanks to xet7 !
This commit is contained in:
1 parent
0d8f8b8d96
commit
3b1c7ce7bb
12 files changed
+187
-166
No files matched your search
@@ -20,82 +20,21 @@ the changelog only says what changed.
|
||||
- The WebKit stage cannot run in the Flatpak sandbox: MiniBrowser needs its own
|
||||
sandbox, and turning that off is not allowed there. Run it on a host.
|
||||
|
||||
## Translations in the wrong language
|
||||
|
||||
Found on 2026-11-20 while translating the new Import and menu headings: these
|
||||
locale files hold values in another language than their tag, which the
|
||||
translation rules say to replace with the tag's language.
|
||||
|
||||
- `se` (Northern Sami): values seeded in Finnish.
|
||||
- `ve-PP` (Veps): values in Venda, for example `board-view-cumulative-flow`.
|
||||
- `uz-AR` (Uzbek, Arabic script): values in Latin script, for example
|
||||
`calendar` as "Kalendar".
|
||||
- `wal` (Wolaytta): values in Ethiopic script, for example "ጊዜ" for time.
|
||||
- `mt` (Maltese): `board-view-cumulative-flow` in Italian.
|
||||
- `nd` (Northern Ndebele): the view names look like Shona.
|
||||
- `hy` (Armenian): `board` as Վարչություն, "administration".
|
||||
|
||||
## Menus grouped so they are easier to understand
|
||||
|
||||
Asked on 2026-11-20, after the Admin Panel was reorganised into groups. The
|
||||
Board View menu and the Import and Export lists are grouped now; this is the
|
||||
rest.
|
||||
|
||||
- Swimlanes view: the swimlane, list and card hamburger menus and the board
|
||||
header's buttons, reviewed against the same idea - entries grouped by what
|
||||
they act on (the item itself, its contents, moving and copying, view and
|
||||
settings, archive and delete), with a heading per group as the board menu
|
||||
already has.
|
||||
|
||||
## Import and Export of large files
|
||||
|
||||
Asked on 2026-11-20. A file chosen on the Import page for the generalized
|
||||
importer's sources is now streamed to the server's disk a part at a time, with
|
||||
the free space checked before and while it is written and the Admin Panel
|
||||
limit (none: the free disk space) applied; zips are opened from the disk and
|
||||
mailboxes read a message at a time. This is the rest.
|
||||
mailboxes read a message at a time; WeKan's own .zip and Trello's package use
|
||||
the same upload, limit and disk check. This is the rest.
|
||||
|
||||
- JSON and text formats are still read whole into memory once uploaded:
|
||||
read them in bounded pieces too (a streaming JSON parser for the large JSON
|
||||
exports - Jira, Taiga, Telegram - and line by line for the CSV and line
|
||||
formats).
|
||||
- WeKan's own .zip and Trello's package have upload routes of their own with
|
||||
fixed caps (200 MB and 1 GB): route them through the same upload, limit and
|
||||
disk check.
|
||||
- Export: write a large board to the response as it is collected, in parts,
|
||||
instead of building the whole file in memory first.
|
||||
|
||||
## Reports to add to Board Views
|
||||
|
||||
Found on 2026-11-20 by comparing the Board View menu
|
||||
(`models/lib/boardViewSettings.js`: Gantt, Burndown, Burnup, Cumulative Flow,
|
||||
Control, Cycle Time, Lead Time, Flow Efficiency, Throughput Histogram, WIP
|
||||
Run, Aging WIP, Blocker Analysis, Monte Carlo, Process Behavior, Size vs.
|
||||
Cycle Time, Velocity, Sprint Report, Dashboard, Pulse, Statistics, Time) with
|
||||
the reports of Jira, Azure DevOps, Nave, ActionableAgile and the Flow
|
||||
Framework. Each is a Board View with the same export bar at its top as the
|
||||
other reports, drawn with Chart.js (already a dependency) from the board's
|
||||
cards and change history.
|
||||
|
||||
- Created vs. Resolved: cards added and cards finished per day or week, and
|
||||
the running difference - is the work arriving faster than it leaves.
|
||||
- Net Flow: departures minus arrivals per week as bars, with the trend.
|
||||
- Time in List: how long cards spent in each list (median and 85th
|
||||
percentile per list), from the change history of their moves.
|
||||
- Average Age: the mean age of the unfinished cards on each day.
|
||||
- Throughput Run Chart: cards finished per week over time, with the median
|
||||
line (the histogram shows only the distribution).
|
||||
- Workload: open cards and their sizes per assignee and member, stacked by
|
||||
list.
|
||||
- Distribution: open and finished cards by label, swimlane, card type or a
|
||||
custom field, as a pie and as a table (Jira's pie chart report, the Flow
|
||||
Framework's flow distribution).
|
||||
- Due Date Performance: finished cards on time, late and without a due date,
|
||||
per week, and how late on average.
|
||||
- Release Burndown: per Scrum release, the remaining work and the projected
|
||||
finish from the velocity of the last sprints.
|
||||
- Epic Progress: for each card with subtasks, its subtasks done, open and
|
||||
added over time.
|
||||
- Dependency Map: the cards and their dependencies (blocks, fixes, relates) as
|
||||
a graph, the blocked chains first.
|
||||
- Activity Heatmap: changes per weekday and hour, and per member and day.
|
||||
- Export: WeKan's own JSON export is written to the response as it is
|
||||
collected (`buildStream`), and the .mbox and .enex exports a page of 500
|
||||
cards at a time (`streamExternalExport`); the other tools' formats are still
|
||||
built whole in memory and then sent. Write those in parts too, where the
|
||||
format allows it.
|
||||
+4
-2
@@ -360,10 +360,12 @@ if (Meteor.isServer) {
|
||||
let user = null;
|
||||
const respond = async () => {
|
||||
await require('/server/lib/adminOnlyCustomFields').assertFieldExport(boardId, user?._id);
|
||||
const fields = parseExportFields(req.query && req.query.fields, BOARD_EXPORT_FIELD_KEYS);
|
||||
// A format of independent parts is written a page of cards at a time.
|
||||
if (await require('/server/lib/renderExternalExport').streamExternalExport(res, boardId, format, fields)) return;
|
||||
// The bytes for this format (server/lib/renderExternalExport.js), shared
|
||||
// with "Export all boards" (server/lib/exportAllBoards.js).
|
||||
const rendered = await require('/server/lib/renderExternalExport').renderExternalExport(boardId, format,
|
||||
parseExportFields(req.query && req.query.fields, BOARD_EXPORT_FIELD_KEYS));
|
||||
const rendered = await require('/server/lib/renderExternalExport').renderExternalExport(boardId, format, fields);
|
||||
if (rendered.json !== undefined) {
|
||||
sendJsonResult(res, { code: 200, data: rendered.json });
|
||||
return;
|
||||
|
||||
+28
-43
@@ -14,7 +14,6 @@ const MAX_IMPORT_DOCUMENT_BYTES = 256 * 1024 * 1024;
|
||||
|
||||
runOnServer(function () {
|
||||
const fs = Npm.require('fs');
|
||||
const os = Npm.require('os');
|
||||
const path = Npm.require('path');
|
||||
const { WebApp } = require('meteor/webapp');
|
||||
const { safeRoute } = require('/server/apiMiddleware');
|
||||
@@ -44,47 +43,33 @@ runOnServer(function () {
|
||||
// and the name is never used as a path - a temp name is generated instead, so
|
||||
// an entry called `../../etc/cron.d/x` can only ever be an attachment called
|
||||
// that.
|
||||
const MAX_ZIP_BYTES = (() => {
|
||||
// The upload is streamed to WRITABLE_PATH/import-uploads by the same receiver
|
||||
// as every Import page upload (server/lib/importUpload.js): the free space is
|
||||
// checked before and while it is written, and the largest import is the
|
||||
// Admin Panel's (Settings / Security), else WEKAN_IMPORT_ZIP_MAX_BYTES when
|
||||
// set, else none - as large as the free disk space lets it finish. Its name
|
||||
// is generated, never the archive's, and the entries below are matched by
|
||||
// name, never used as paths.
|
||||
const receiveToTempFile = async req => {
|
||||
const { receiveUpload } = require('/server/lib/importUpload');
|
||||
const { importLimitBytes } = require('/server/lib/importDiskSpace');
|
||||
const Settings = require('/models/settings').default;
|
||||
const configured = parseInt(process.env.WEKAN_IMPORT_ZIP_MAX_BYTES, 10);
|
||||
return Number.isFinite(configured) && configured > 0 ? configured : 5 * 1024 * 1024 * 1024;
|
||||
})();
|
||||
|
||||
const receiveToTempFile = req => new Promise((resolve, reject) => {
|
||||
// The name is ours, not the archive's - but it is still built with
|
||||
// safeEntryPath, which resolves the result and requires it to be BENEATH the
|
||||
// temp directory. A generated name that cannot escape is worth as much as
|
||||
// the check that proves it, and tests/fixedVulnerabilityClasses.test.cjs
|
||||
// asks every file that reads an archive to show one (ZipBleed).
|
||||
const tempPath = safeEntryPath(os.tmpdir(),
|
||||
[`wekan-import-${Date.now()}-${Math.random().toString(36).slice(2)}.zip`]);
|
||||
if (!tempPath) {
|
||||
reject(new Error('import-temp-path'));
|
||||
return;
|
||||
const limit = importLimitBytes(await Settings.findOneAsync({}, { fields: { importMaxMegabytes: 1 } }))
|
||||
|| (Number.isFinite(configured) && configured > 0 ? configured : 0);
|
||||
try {
|
||||
const { file, size } = await receiveUpload(req, { userId: 'import-zip', source: 'wekan', limit });
|
||||
// The generated name is still proven to be beneath the upload directory
|
||||
// (ZipBleed: tests/fixedVulnerabilityClasses.test.cjs).
|
||||
const tempPath = safeEntryPath(require('path').dirname(file), [require('path').basename(file)]);
|
||||
if (!tempPath) throw new Error('import-temp-path');
|
||||
return { tempPath, received: size };
|
||||
} catch (error) {
|
||||
if (error && error.code === 'import-too-large') throw new Error('import-zip-too-large');
|
||||
if (error && error.code === 'import-disk-full') throw new Error('import-disk-full');
|
||||
throw error;
|
||||
}
|
||||
const out = fs.createWriteStream(tempPath);
|
||||
let received = 0;
|
||||
let aborted = false;
|
||||
|
||||
const fail = error => {
|
||||
if (aborted) return;
|
||||
aborted = true;
|
||||
try { req.destroy(); } catch (e) { /* already gone */ }
|
||||
try { out.destroy(); } catch (e) { /* already gone */ }
|
||||
fs.promises.unlink(tempPath).catch(() => {});
|
||||
reject(error);
|
||||
};
|
||||
|
||||
req.on('data', chunk => {
|
||||
received += chunk.length;
|
||||
// A cap that is checked as it arrives, not after: the point of streaming
|
||||
// is that an oversized upload never lands.
|
||||
if (received > MAX_ZIP_BYTES) fail(new Error('import-zip-too-large'));
|
||||
});
|
||||
req.on('error', fail);
|
||||
out.on('error', fail);
|
||||
out.on('finish', () => (aborted ? undefined : resolve({ tempPath, received })));
|
||||
req.pipe(out);
|
||||
});
|
||||
};
|
||||
|
||||
/**
|
||||
* @operation importZip
|
||||
@@ -239,11 +224,11 @@ runOnServer(function () {
|
||||
answer(400, { error: error.reason });
|
||||
return;
|
||||
}
|
||||
const message = error && error.message === 'import-zip-too-large'
|
||||
? 'import-zip-too-large'
|
||||
const message = error && ['import-zip-too-large', 'import-disk-full'].includes(error.message)
|
||||
? error.message
|
||||
: 'import-failed';
|
||||
console.error('importZip failed', error);
|
||||
answer(message === 'import-zip-too-large' ? 413 : 500, { error: message });
|
||||
answer(message === 'import-zip-too-large' ? 413 : message === 'import-disk-full' ? 507 : 500, { error: message });
|
||||
} finally {
|
||||
if (tempPath) fs.promises.unlink(tempPath).catch(() => {});
|
||||
}
|
||||
|
||||
@@ -102,8 +102,15 @@ function enml(item) {
|
||||
+ `<en-note>${blocks.join('')}</en-note>`;
|
||||
}
|
||||
|
||||
export function formatEnex({ items = [] } = {}, { now = new Date() } = {}) {
|
||||
const notes = items.map(item => {
|
||||
// The parts of an .enex, so a large board can be written a page at a time
|
||||
// (server/lib/renderExternalExport.js): the head, the notes of some cards, the
|
||||
// tail. formatEnex is the three together.
|
||||
export const enexHead = (now = new Date()) => '<?xml version="1.0" encoding="UTF-8"?>\n'
|
||||
+ '<!DOCTYPE en-export SYSTEM "http://xml.evernote.com/pub/evernote-export4.dtd">\n'
|
||||
+ `<en-export export-date="${toEnexDate(now)}" application="WeKan" version="1">\n`;
|
||||
export const ENEX_TAIL = '</en-export>\n';
|
||||
export function enexNotes(items = []) {
|
||||
return items.map(item => {
|
||||
const created = toEnexDate(item.createdAt);
|
||||
const attributes = [
|
||||
item.dueAt ? `<reminder-time>${toEnexDate(item.dueAt)}</reminder-time>` : '',
|
||||
@@ -120,8 +127,9 @@ export function formatEnex({ items = [] } = {}, { now = new Date() } = {}) {
|
||||
attributes ? ` <note-attributes>${attributes}</note-attributes>` : '',
|
||||
' </note>',
|
||||
].filter(Boolean).join('\n');
|
||||
});
|
||||
return '<?xml version="1.0" encoding="UTF-8"?>\n'
|
||||
+ '<!DOCTYPE en-export SYSTEM "http://xml.evernote.com/pub/evernote-export4.dtd">\n'
|
||||
+ `<en-export export-date="${toEnexDate(now)}" application="WeKan" version="1">\n${notes.join('\n')}\n</en-export>\n`;
|
||||
}).map(note => `${note}\n`).join('');
|
||||
}
|
||||
|
||||
export function formatEnex({ items = [] } = {}, { now = new Date() } = {}) {
|
||||
return enexHead(now) + enexNotes(items) + ENEX_TAIL;
|
||||
}
|
||||
@@ -59,11 +59,13 @@ async function moveCompletionRules(boardId) {
|
||||
return out;
|
||||
}
|
||||
|
||||
async function collect(boardId, fields, format) {
|
||||
// cardIds: only these cards (one page of a streamed export), else all.
|
||||
async function collect(boardId, fields, format, cardIds = null) {
|
||||
const board = await ReactiveCache.getBoard(boardId);
|
||||
const lists = await ReactiveCache.getLists({ boardId, archived: false }, { sort: { sort: 1 } });
|
||||
const swimlanes = await ReactiveCache.getSwimlanes({ boardId, archived: false }, { sort: { sort: 1 } });
|
||||
const cards = await ReactiveCache.getCards({ boardId, archived: false }, { sort: { sort: 1 } });
|
||||
const cards = await ReactiveCache.getCards({ boardId, archived: false, ...(cardIds ? { _id: { $in: cardIds } } : {}) },
|
||||
{ sort: { sort: 1 } });
|
||||
const listById = {};
|
||||
const listRecords = new Map(lists.map(list => [list._id, list]));
|
||||
lists.forEach(l => { listById[l._id] = l.title; });
|
||||
@@ -173,6 +175,21 @@ async function collect(boardId, fields, format) {
|
||||
items: items.map(item => gateItem(item, wanted)) };
|
||||
}
|
||||
|
||||
// A text export a page of cards at a time (server/lib/renderExternalExport.js
|
||||
// streams the formats that are a sequence of independent parts): the board's
|
||||
// card ids are read first, then each page is collected and passed through the
|
||||
// export boundary as the whole document is, so only one page is in memory.
|
||||
export async function* externalExportPages(boardId, format, fields, pageSize = 500) {
|
||||
const { secureTransfer } = require('/server/lib/secureTransfer');
|
||||
const context = { direction: 'export', source: `export:${format}` };
|
||||
const ids = (await ReactiveCache.getCards({ boardId, archived: false }, { sort: { sort: 1 }, fields: { _id: 1 } })).map(c => c._id);
|
||||
for (let at = 0; at < ids.length; at += pageSize) {
|
||||
// eslint-disable-next-line no-await-in-loop
|
||||
const collected = await collect(boardId, fields, format, ids.slice(at, at + pageSize));
|
||||
yield { ...collected, items: secureTransfer(collected.items, context) };
|
||||
}
|
||||
}
|
||||
|
||||
export async function buildExternalExport(boardId, format, fields) {
|
||||
const formatter = formatters[format];
|
||||
if (!formatter) return null;
|
||||
|
||||
@@ -48,4 +48,41 @@ async function renderExternalExport(boardId, format, fields) {
|
||||
return { json: built };
|
||||
}
|
||||
|
||||
module.exports = { renderExternalExport };
|
||||
// The text formats that are a sequence of independent parts, written to the
|
||||
// response a page of cards at a time instead of built whole: an .mbox is one
|
||||
// message after another, an .enex a head, notes and a tail.
|
||||
const STREAMED = {
|
||||
mbox: () => {
|
||||
const { formatMbox } = require('/models/lib/mailFormat');
|
||||
let first = true;
|
||||
return { head: '', page: collected => {
|
||||
const text = formatMbox(collected);
|
||||
const out = first || !text ? text : `\n${text}`;
|
||||
first = false;
|
||||
return out;
|
||||
}, tail: '' };
|
||||
},
|
||||
enex: () => {
|
||||
const { enexHead, enexNotes, ENEX_TAIL } = require('/models/lib/enexFormat');
|
||||
return { head: enexHead(), page: collected => enexNotes(collected.items), tail: ENEX_TAIL };
|
||||
},
|
||||
};
|
||||
|
||||
// Writes the export to `res` page by page, or answers false when the format
|
||||
// is not one that streams.
|
||||
async function streamExternalExport(res, boardId, format, fields) {
|
||||
if (!STREAMED[format]) return false;
|
||||
const { externalExportPages } = require('/models/lib/externalExporters');
|
||||
const parts = STREAMED[format]();
|
||||
res.writeHead(200, { 'Content-Type': `${TEXT_TYPES[format]}; charset=utf-8` });
|
||||
if (parts.head) res.write(parts.head);
|
||||
for await (const collected of externalExportPages(boardId, format, fields)) {
|
||||
const text = parts.page(collected);
|
||||
// Wait for the socket when it is full, so the pages do not pile up in memory.
|
||||
if (text && !res.write(text)) await new Promise(resolve => res.once('drain', resolve));
|
||||
}
|
||||
res.end(parts.tail);
|
||||
return true;
|
||||
}
|
||||
|
||||
module.exports = { renderExternalExport, streamExternalExport, STREAMED };
|
||||
@@ -24,8 +24,10 @@ import {
|
||||
// guards count what streams, so nothing is held in memory but the board JSON.
|
||||
|
||||
// --- zip-bomb / abuse guards -------------------------------------------------
|
||||
const MAX_ZIP_BYTES = 200 * 1024 * 1024; // 200 MB compressed upload cap
|
||||
const MAX_TOTAL_UNCOMPRESSED = 1024 * 1024 * 1024; // 1 GB total uncompressed
|
||||
// The upload and what its entries inflate to are bounded by the largest
|
||||
// import of Admin Panel / Settings / Security, or with none set by the free
|
||||
// disk space (server/lib/importUpload.js, importDiskSpace.js) - no fixed cap.
|
||||
const DEFAULT_TOTAL_UNCOMPRESSED = 1024 * 1024 * 1024; // when the free space cannot be measured
|
||||
const MAX_ENTRIES = 5000; // maximum number of files inside the zip
|
||||
const MAX_FILE_BYTES = 100 * 1024 * 1024; // 100 MB per extracted file
|
||||
|
||||
@@ -172,7 +174,8 @@ async function importSingleBoard(payload, userId) {
|
||||
}
|
||||
|
||||
// Read and import every board in the zip file, returning { boardIds, errors }.
|
||||
async function importZipFile(zipPath, userId, membersMode) {
|
||||
async function importZipFile(zipPath, userId, membersMode, maxUncompressed = DEFAULT_TOTAL_UNCOMPRESSED) {
|
||||
const MAX_TOTAL_UNCOMPRESSED = maxUncompressed;
|
||||
const unzipper = require('unzipper');
|
||||
const zip = await unzipper.Open.file(zipPath);
|
||||
|
||||
@@ -336,43 +339,29 @@ Meteor.startup(() => {
|
||||
// A .zip package: the body-parser does not touch application/zip, so the
|
||||
// raw stream is intact. It is written to a temporary file as it arrives,
|
||||
// under the same compressed-upload cap, and opened from there.
|
||||
const fs = require('fs');
|
||||
const os = require('os');
|
||||
const path = require('path');
|
||||
const { Random } = require('meteor/random');
|
||||
const tempPath = path.join(os.tmpdir(), `wekan-trello-import-${Random.id()}.zip`);
|
||||
const Settings = require('/models/settings').default;
|
||||
const { receiveUpload, removeUpload } = require('/server/lib/importUpload');
|
||||
const { importLimitBytes, roomFor } = require('/server/lib/importDiskSpace');
|
||||
const limit = importLimitBytes(await Settings.findOneAsync({}, { fields: { importMaxMegabytes: 1 } }));
|
||||
const membersMode = (req.query && req.query.membersMode) || '';
|
||||
let total = 0;
|
||||
let aborted = false;
|
||||
const out = fs.createWriteStream(tempPath);
|
||||
const cleanup = () => fs.promises.unlink(tempPath).catch(() => {});
|
||||
const fail = (code, error) => {
|
||||
if (aborted) return;
|
||||
aborted = true;
|
||||
try { req.unpipe(out); out.destroy(); } catch (e) { /* closed */ }
|
||||
cleanup();
|
||||
sendJson(code, { error });
|
||||
};
|
||||
req.on('data', chunk => {
|
||||
total += chunk.length;
|
||||
if (total > MAX_ZIP_BYTES) {
|
||||
fail(413, 'import-trello-zip-too-large');
|
||||
req.destroy();
|
||||
}
|
||||
});
|
||||
req.on('error', () => fail(400, 'import-trello-zip-read-failed'));
|
||||
out.on('error', () => fail(500, 'import-trello-zip-read-failed'));
|
||||
out.on('finish', async () => {
|
||||
if (aborted) return;
|
||||
try {
|
||||
const result = await importZipFile(tempPath, userId, membersMode);
|
||||
sendJson(200, result);
|
||||
} catch (e) {
|
||||
sendJson(400, { error: (e && e.message) || 'import-trello-zip-failed' });
|
||||
} finally {
|
||||
cleanup();
|
||||
}
|
||||
});
|
||||
req.pipe(out);
|
||||
let upload;
|
||||
try {
|
||||
upload = await receiveUpload(req, { userId, source: 'trello', limit });
|
||||
} catch (e) {
|
||||
const code = e && e.code;
|
||||
sendJson(code === 'import-too-large' ? 413 : code === 'import-disk-full' ? 507 : 400,
|
||||
{ error: code === 'import-too-large' ? 'import-trello-zip-too-large' : code || 'import-trello-zip-read-failed' });
|
||||
return;
|
||||
}
|
||||
try {
|
||||
const room = await roomFor(require('path').dirname(upload.file));
|
||||
const inflate = limit || (Number.isFinite(room) ? room : DEFAULT_TOTAL_UNCOMPRESSED);
|
||||
const result = await importZipFile(upload.file, userId, membersMode, inflate);
|
||||
sendJson(200, result);
|
||||
} catch (e) {
|
||||
sendJson(400, { error: (e && e.message) || 'import-trello-zip-failed' });
|
||||
} finally {
|
||||
await removeUpload(upload.uploadId);
|
||||
}
|
||||
});
|
||||
});
|
||||
@@ -450,7 +450,9 @@ test('every downloadable format carries the selection, and the server reads it',
|
||||
}
|
||||
// The route hands the selection to the shared renderer
|
||||
// (server/lib/renderExternalExport.js), which hands it to the collector.
|
||||
assert.ok(/renderExternalExport\(boardId, format,\s*\n\s*parseExportFields\(req\.query && req\.query\.fields, BOARD_EXPORT_FIELD_KEYS\)\)/
|
||||
// The selection is parsed once and given to both the streamed path (.mbox,
|
||||
// .enex, a page at a time) and the shared renderer.
|
||||
assert.ok(/const fields = parseExportFields\(req\.query && req\.query\.fields, BOARD_EXPORT_FIELD_KEYS\);[\s\S]{0,300}streamExternalExport\(res, boardId, format, fields\)[\s\S]{0,300}renderExternalExport\(boardId, format, fields\)/
|
||||
.test(read('models/export.js')),
|
||||
'and the external-tool exports (Trello, Jira, GitHub, Markdown, ...) must read it too');
|
||||
});
|
||||
|
||||
@@ -69,3 +69,24 @@ test('a board exported as .enex reads back as the same notes', () => {
|
||||
assert.deepStrictEqual(card.tags, ['shop', 'To Do']);
|
||||
assert.strictEqual(card.date_due, '2026-11-20T09:00:00.000Z');
|
||||
});
|
||||
|
||||
test('a large board is exported a page at a time: the pages together are the whole file', async () => {
|
||||
const mail = await import('../models/lib/mailFormat.js');
|
||||
const items = Array.from({ length: 5 }, (_, i) => ({ cardId: `c${i}`, title: `Card ${i}`, description: `Text ${i}`,
|
||||
creator: 'anna', createdAt: `2026-11-0${i + 1}T10:00:00.000Z`, comments: i % 2 ? [{ text: 'ok', author: 'matti' }] : [] }));
|
||||
const now = new Date('2026-11-16T10:00:00Z');
|
||||
const pages = [items.slice(0, 2), items.slice(2, 4), items.slice(4)];
|
||||
assert.strictEqual(enex.enexHead(now) + pages.map(p => enex.enexNotes(p)).join('') + enex.ENEX_TAIL,
|
||||
enex.formatEnex({ items }, { now }), 'an .enex: head, the notes page by page, tail');
|
||||
assert.strictEqual(pages.map(p => mail.formatMbox({ items: p })).join('\n'), mail.formatMbox({ items }),
|
||||
'an .mbox: the messages page by page');
|
||||
const fs = require('node:fs');
|
||||
const path = require('node:path');
|
||||
const read = rel => fs.readFileSync(path.join(__dirname, '..', rel), 'utf8');
|
||||
const render = read('server/lib/renderExternalExport.js');
|
||||
assert.match(render, /for await \(const collected of externalExportPages\(boardId, format, fields\)\)/);
|
||||
assert.match(render, /if \(text && !res\.write\(text\)\) await new Promise\(resolve => res\.once\('drain', resolve\)\);/,
|
||||
'it waits for the socket, so pages do not pile up in memory');
|
||||
assert.match(read('models/lib/externalExporters.js'), /export async function\* externalExportPages\(boardId, format, fields, pageSize = 500\)/);
|
||||
assert.match(read('models/export.js'), /streamExternalExport\(res, boardId, format, fields\)\) return;/);
|
||||
});
|
||||
@@ -135,7 +135,9 @@ async function main() {
|
||||
assert.doesNotMatch(route, /buildKanboardExport|\/export\/kanboard', safeRoute/, 'one Kanboard route, the shared one');
|
||||
// The route checks the fields, then renders through the shared renderer
|
||||
// (server/lib/renderExternalExport.js), which calls the collector.
|
||||
assert.match(route, /await require\('\/server\/lib\/adminOnlyCustomFields'\)\.assertFieldExport\(boardId, user\?\._id\);[\s\S]{0,240}renderExternalExport\(boardId, format,/);
|
||||
// The fields are checked before either path: the streamed one (.mbox, .enex)
|
||||
// and the shared renderer.
|
||||
assert.match(route, /await require\('\/server\/lib\/adminOnlyCustomFields'\)\.assertFieldExport\(boardId, user\?\._id\);[\s\S]{0,240}streamExternalExport\(res, boardId, format, fields\)[\s\S]{0,300}renderExternalExport\(boardId, format,/);
|
||||
assert.match(read('server/lib/renderExternalExport.js'), /const built = await buildExternalExport\(boardId, format, fields\);/);
|
||||
console.log(' ok - extras are selected by the export selection; Kanboard uses the shared route');
|
||||
}
|
||||
|
||||
@@ -153,3 +153,17 @@ test('Admin Panel / Settings / Security sets the largest import; empty is no lim
|
||||
assert.strictEqual(typeof en[key], 'string', key);
|
||||
}
|
||||
});
|
||||
|
||||
test('WeKan\'s own .zip and Trello\'s package use the same upload, limit and disk check', () => {
|
||||
const read = rel => fs.readFileSync(path.join(__dirname, '..', rel), 'utf8');
|
||||
const zip = read('models/importZip.js');
|
||||
assert.match(zip, /receiveUpload\(req, \{ userId: 'import-zip', source: 'wekan', limit \}\)/);
|
||||
assert.match(zip, /importLimitBytes\(await Settings\.findOneAsync/);
|
||||
assert.match(zip, /message === 'import-disk-full' \? 507/);
|
||||
assert.doesNotMatch(zip, /5 \* 1024 \* 1024 \* 1024/, 'no fixed 5 GB cap: the limit, else the free space');
|
||||
const trello = read('server/routes/importTrelloZip.js');
|
||||
assert.match(trello, /receiveUpload\(req, \{ userId, source: 'trello', limit \}\)/);
|
||||
assert.match(trello, /importZipFile\(upload\.file, userId, membersMode, inflate\)/);
|
||||
assert.doesNotMatch(trello, /const MAX_ZIP_BYTES = 200/, 'no fixed 200 MB cap');
|
||||
assert.match(trello, /await removeUpload\(upload\.uploadId\)/, 'the upload is removed once imported');
|
||||
});
|
||||
@@ -239,7 +239,10 @@ test('a .zip is UPLOADED, not unpacked in the browser', () => {
|
||||
|
||||
test('the upload route never holds the archive in memory', () => {
|
||||
const route = read('models/importZip.js');
|
||||
assert.ok(/req\.pipe\(out\)/.test(route), 'the body is streamed to a temp file');
|
||||
// Through the shared receiver since 2026-11-20 (server/lib/importUpload.js),
|
||||
// which pipes the request to its file with the limit and disk checks.
|
||||
assert.ok(/receiveUpload\(req, /.test(route), 'the body is streamed to a temp file');
|
||||
assert.ok(/req\.pipe\(out\)/.test(read('server/lib/importUpload.js')), 'by the shared receiver, a part at a time');
|
||||
assert.ok(/unzipper\.Open\.file\(tempPath\)/.test(route),
|
||||
'and the archive is read through its central directory, entry by entry');
|
||||
assert.ok(/entry\.stream\(\)/.test(route), 'each attachment is opened as a stream');
|
||||
@@ -250,7 +253,9 @@ test('the upload route never holds the archive in memory', () => {
|
||||
|
||||
test('an oversized upload is refused as it arrives (negative)', () => {
|
||||
const route = read('models/importZip.js');
|
||||
assert.ok(/received > MAX_ZIP_BYTES/.test(route), 'the cap is checked per chunk');
|
||||
// The shared receiver checks the limit per chunk (server/lib/importUpload.js).
|
||||
assert.ok(/if \(limit && total > limit\)/.test(read('server/lib/importUpload.js')), 'the cap is checked per chunk');
|
||||
assert.ok(/receiveUpload\(req, \{ userId: 'import-zip', source: 'wekan', limit \}\)/.test(route), 'with the zip route\'s limit');
|
||||
assert.ok(/WEKAN_IMPORT_ZIP_MAX_BYTES/.test(route), 'and is configurable');
|
||||
assert.ok(/import-zip-too-large/.test(route), 'with an answer that says which limit');
|
||||
});
|
||||
|
||||
Reference in new issue
Block a user