Recognize supplementary (non-BMP) punctuations around emphasis delimiter runs

[tats-u]

markdig/src/Markdig/Helpers/CharHelper.cs

Lines 37 to 60 in fb3fe8b

	public static void CheckOpenCloseDelimiter(char pc, char c, bool enableWithinWord, out bool canOpen, out bool canClose)
	{
	pc.CheckUnicodeCategory(out bool prevIsWhiteSpace, out bool prevIsPunctuation);
	c.CheckUnicodeCategory(out bool nextIsWhiteSpace, out bool nextIsPunctuation);
	var prevIsExcepted = prevIsPunctuation && IsPunctuationException(pc);
	var nextIsExcepted = nextIsPunctuation && IsPunctuationException(c);
	// A left-flanking delimiter run is a delimiter run that is
	// (1) not followed by Unicode whitespace, and either
	// (2a) not followed by a punctuation character or
	// (2b) followed by a punctuation character and preceded by Unicode whitespace or a punctuation character.
	// For purposes of this definition, the beginning and the end of the line count as Unicode whitespace.
	canOpen = !nextIsWhiteSpace &&
	((!nextIsPunctuation || nextIsExcepted) || prevIsWhiteSpace || prevIsPunctuation);
	// A right-flanking delimiter run is a delimiter run that is
	// (1) not preceded by Unicode whitespace, and either
	// (2a) not preceded by a punctuation character, or
	// (2b) preceded by a punctuation character and followed by Unicode whitespace or a punctuation character.
	// For purposes of this definition, the beginning and the end of the line count as Unicode whitespace.
	canClose = !prevIsWhiteSpace &&
	((!prevIsPunctuation || prevIsExcepted) || nextIsWhiteSpace || nextIsPunctuation);

• Recognize non-BMP punctuation & symbols commonmark/commonmark.js#297
• Fix Unicode handling in emphasis parsing (flanking rules) commonmark/commonmark-java#198
• Add supplementary (non-BMP) currency symbol in Unicode symbol example commonmark/commonmark-spec#794

The specs says "A character is a Unicode code point." However, the char type is just a UTF-16 code unit, which does not cover supplementary characters (whose code points are U+100000 or greater).

https://spec.commonmark.org/0.31.2/#character

[tats-u]

The problems in System.Text.Rune:

• does not accept surrogate code points (U+D800–U+DFFF) because it is for Unicode scalar values (surrogate code points are out of their range).
• Exclusive for .NET (unavailable in .NET Framework)
• No official NuGet backport package for .NET Standard 2.0

[MihaZupan]

Rune does support surrogates, you can do new Rune(high, low). It's also thankfully one of those types that's trivial to polyfill.

If limited to e.g. EmphasisInline, supporting non-BMP punctuations seems fine to me.

[tats-u]

Rune does support surrogates, you can do new Rune(high, low).

It's how we get a supplementary code point from a well-formed surrogate pair. What I mean is how to handle an ill-formed isolated surrogate code unit.

If limited to e.g. EmphasisInline, supporting non-BMP punctuations seems fine to me.

Sufficient.

[tats-u]

https://learn.microsoft.com/en-US/dotnet/api/system.globalization.charunicodeinfo.getunicodecategory?view=net-8.0#system-globalization-charunicodeinfo-getunicodecategory(system-int32)

Another problem is the CharUnicodeInfo.GetUnicodeCategory has an overload that takes int but it's exclusive to .NET.

Should we substitute the (string, int) overload for that in .NET Standard 2.0?

[MihaZupan]

how to handle an ill-formed isolated surrogate code unit.

I'm not sure I understand the concern. If we happen to see a valid surrogate pair we can check its category. If it's not a valid pair the behavior stays the same - it's currently treated as an arbitrary uninteresting char.

Another problem is the CharUnicodeInfo.GetUnicodeCategory has an overload that takes int but it's exclusive to .NET.

That's okay, older runtimes just have to deal with being slow :)
That's how Rune.GetUnicodeCategory is imlpemented as well.

[tats-u]

If it's not a valid pair the behavior stays the same - it's currently treated as an arbitrary uninteresting char.

Will be acceptable because the Unicode category of all surrogate code points is Cs—i.e. not P or S.

It looks like you prefer Rune to int or unit—how about Rune? (nullable)? Rune is a struct, so its nullable type is fully supported in older C# or .NET Standard 2.0. If the adjacent code point is an isolated surrogate one, the value should be null. (Falling back to another value expressing a non-punctuation scalar vale is fine though)

[tats-u]

Probably some methods that return Rune, int, or uint need to be added in StringSlice.

[tats-u]

memo: how to call internal unsafe static UnicodeCategory InternalGetUnicodeCategory(int ch) in .NET Framework via reflection:

using System;
using System.Reflection;
using System.Globalization;

var method = typeof(CharUnicodeInfo).GetMethod(
            "InternalGetUnicodeCategory",
            BindingFlags.NonPublic | BindingFlags.Static,
            null,
            new Type[] { typeof(int) }
            null);
if (method is not null)
{
    var getUnicodeCategory = (Func<int, UnicodeCategory>)method.CreateDelegate(
                    typeof(Func<int, UnicodeCategory>));
}