Project pages

[jace]

Projects need a new model named Page. This will be the third content type in a project after Proposal (third party submission) and Update (editor's blog), and addresses the gaps of the existing two. Characteristics:

1. Pages have wiki-like URL stubs, without a UUID, for nicer URLs
2. Renames are tracked with a PageRedirect model
3. Pages do not have a timestamp or byline
4. Pages do not accept comments. Should a discussion be necessary, pages should start as submissions. The semantics of how a discussion is conducted and converted into a are outside the scope of this ticket
5. Page are project content, editable by the Editor and Promoter roles

Nice to have (check-off when these features are implemented in later PRs):

1. Pages should support wiki-links between pages using the [[Page name]] syntax. There will be an implementation detail to convert from relative URLs to absolute URLs based on where the page content appears
2. As shared-edit content, pages need version history in a PageRevision model
3. Pages also need revision control, allowing for an unpublished draft revision later than a published revision. There has been prior work on achieving this in Eventframe and Hasmail
4. The Proposal.description field should be migrated into an index Page, thereby acquiring all these nice features
5. Pages should support Mustache templating, with a controlled vocabulary that lets a page reflect project state. The index page may want to include a timestamp of the next session, for example

Original text of this issue ticket:

Proposals were renamed to Submissions in December 2020, to switch to a more generic term. However, this is not generic enough for projects with editor-only content. They need:

1. "Page" is a more appropriate term (or "Chapter" or "Section")
2. The Submissions tab is unnecessary. The table of contents can be on the main page
3. "Project overview" is confusing because it doesn't seem like a project. Just "Overview" should do
4. Pages need to be in editor-defined order. This functionality is coming with Proposal sorting refactor #1028
5. Some pages may be short and interesting enough to show expanded, with full content instead of a card (either reuse the Featured tag, or add a new tag)
6. Pages should be groupable into sections, perhaps using labels, in which case labels will also need sorting order
7. All editors should be able to edit all submissions
8. Only editors may submit; CfP is not open to non-editors

While there has been a long-pending desire for wiki-like pages, that remains on the wishlist and is not being implemented here. Wiki-style [[page title]] links remain unrealized.

The following comments are responding to this original text and not the newer Page spec.

[jace]

Possible resolutions:

1. Always allow editors to make submissions without a public CfP
2. A submission may be marked as project content (a) if submitted by an editor, and (b) by the submitter only. This makes it a "page", stored in a new column Proposal.is_page. All renders for submission vs page must filter on is_page. Pages are editable by all editors
3. Pages are displayed without a byline
4. The Submissions tab is hidden when there are no submissions (Proposal.is_page = False)

[jace]

Should pages allow comments or not? Open question. (Decision: NO)

[jace]

There are three distinct expectations here:

1. Submission: a document submitted by an outsider, for the perusal of editors. It should be clear to the reader that this content is not endorsed by the project.
2. Page: project content. Has no attribution or timestamp. Should not have comments, because it's not a discussion item.
3. Post/Chapter: a submission by an editor, implicitly endorsed by the editors.

A post can be represented as a workflow state: Draft, Submission, Post, etc. These new states replace the existing workflow states (like Confirmed) that are better represented by labels. A submission can be converted into a post and vice versa.

A page can also be such a workflow state, or it can be an independent flag. TBD.

[jace]

Updates have visibility state options of public and restricted (participants only). This flag also needs to be available on Session and Proposal.

However, on Proposal it still has the problem of deciding who gets access to set this flag: editors or proposers.

[jace]

"Post" in the sense of a blog post is the same thing as an Update. We could add comments to updates instead of stretching the definition of a Proposal.

Similarly, Page should possibly be a separate data type because it's functionality diverges from Proposal. Pages have URL slugs and arguably should also allow Mustache templating so they can communicate the state of a project (number of submissions, etc).

[jace]

End of comment history for the original issue ticket. We've established Page as a distinct content type.

[jace]

Possible Page models with revisioning:

Page:

• id, primary_key
• project_id, Project this page is in
• No name, title or body, all of which are in the revision model
• Has other stable metadata (like created_at) and incoming foreign keys (can't ignore this!)

PageRevision:

• (page_id, revision_id), compound primary key with foreign key to Page
• user_id, User who created this revision
• name, URL stub
• title, Page title
• body, Content
• revises_id, Link to previous revision that this is based off (compound fkey using page_id, revises_id)
• is_published: bool, indicates the visible page
• is_head: bool, indicates latest revision (there could be multiple heads multi-head rejected, see below)
• is_deleted: bool, Indicates a deleted page or revision
• Conditional unique index, to enforce a limit of one active revision: (project_id, name) where is_published=True
• Check constraint, is_published and is_deleted cannot be True at the same time (rejected, see below on deleting draft head)

To retrieve a page given a URL, query PageRevision.join(Page) on (Page.project=project, PageRevision.name=url-stub), order by (is_published=True before is_published=False, then is_head=True before is_head=False) and get first()

1. If the retrieved item is None, 404. This may be rendered as a prompt to editors to make a page.

2. If the retrieved item has is_deleted == True, render 410 Gone

3. If the retrieved item has is_published == True, we're all set. Render it.

4. If the retrieved item has is_published == False, is_head == True, this page was never published. Decide to render based on app logic: are drafts visible to (a) editors-only, (b) participants, (c) all?

5. If the retrieved item has is_published == False, is_head == False, we have a rename on our hands. The page_id is relevant now. Repeat the query on PageRevision with (project=project, page_id=retrieved.page_id), same ordering, and re-run these steps.

This spec is missing a clean way to edit, because there are two separate concerns here:

1. An unpublished head revision, for collaborators to review and agree on publishing.
2. An ongoing edit session in the browser that must have autosave, and must allow for the same revision to be edited safely, possibly from multiple tabs or devices and by multiple users. Safe editing can be considered a distinct concern from the audit trail of revisions, and should be handled separately.

[jace]

If it's okay to have multiple revision heads, then they also need to be labeled. Options:

1. Allow one head revision per user, creating a conditional unique constraint on (pageid, user_id, where is_head=True). However, this complicates review and publishing, so REJECT.
2. Make is_head a string field, with a unique constraint on (pageid, is_head). However, this field proxies for a changelog, making the constraint untenable for UX. There is also no facility here for a merge revision that combines heads, and we don't want additional complications, so REJECT.
3. Don't allow multiple heads. Apply the same constraint as used on is_published. This is reasonable under current expectations, so ACCEPT.

[jace]

Editing across devices will require a three-way merge. This has to be implemented in a PageDraft model on the client side:

• page_id, revision_id -> PageRevision.(page_id, revision_id), the revision being targeted
• reference_title, reference_body, a copy of the revision's title and body
• reference_timestamp, for the revision's content
• edit_title, edit_body, the widgets in the form

When the client is ready to autosave, it sends the edited content along with the reference timestamp.

• If the timestamp matches, it is saved into the revision. A SQL trigger must verify the timestamp and ensure a parallel save cannot happen. Triggers can't receive variables, so this will require a transient timestamp column.
• If the server has a mismatch, it rejects the save and sends back the new reference title, body and timestamp.
• The client then does a diff on this new reference against its existing reference, and applies the patch on the edit. It then updates the reference and attempts to submit the edit again, repeating the loop as many times as necessary.

This draft model cannot be implemented server-side because it cannot be bound to an edit form. The user may abandon the edit or may open a second tab, causing a connection to the server that is indistinguishable from the first. It may be possible to achieve it by (a) inserting a random draft identifier into the form and (b) removing the draft if unused in a timeout period (say 1 day, to account for the client's intermittent connectivity).

Update: this scheme is has a name and a proper spec: differential synchronization. What we call "reference" here is called "shadow" there, and a proper implementation needs a shadow in both the client and the server.

Additional constraints:

• A page revision is immutable (name, title and body fields) when is_published=True or is_deleted=True or is_head=False. Only the head can be edited, and only when it's not published.
• PageRevision should not have incoming foreign keys (apart from its own revisions), for example from a secondary join table for Label. These secondaries must be cloned with each revision, causing overheads. It is cleaner to have MediaWiki-like markup within the main body (or additional field), and to use that to revise secondaries on the Page model when a revision is published or deleted.

[jace]

Revising the Page models for a single-head scenario:

Page:

• id, primary_key
• project_id, Project this page is in
• published_revision_id, fkey to PageRevision
• head_revision_id, fkey to PageRevision
• is_deleted: bool
• Both fkeys are compound with id because of PageRevision's structure

PageRevision:

• (page_id, page_revision_id): fkey and compound pkey
• name, title, body: Page content
• revises_id: compound fkey to self

In this model, a URL is still accessed by looking up via PageRevision, but the access logic is simpler. However, we have previously had trouble with bi-directional fkeys in Organization <-> Team, back when there were mandatory Owners and Members teams. They make SQL backup/restore hard. If this is an okay risk, this approach may have better performance.

[jace]

More considerations:

1. Don't bother with editable drafts. Keep the simple edit form (open, edit, save), and simply take a snapshot of the previous content for version history.

2. Reversing this, let there be a permanent editable draft, and take snapshots to represent published revisions. Default view will render the latest published revision, falling back to the draft.

3. Support for "edit requests". For eg, if we apply this revision control tech to Proposal, we have a product requirement where an editor corrects a submission and asks the proposer to accept or decline the edit. This will require per-user multi-head revisions.

[jace]

Project page names should be based on the stable URL generator described in hasgeek/coaster#301. This is required for wiki links.