Markdown reader: use CommonMark rules for list item nesting.

Closes #3511.

Previously pandoc used the four-space rule: continuation paragraphs,
sublists, and other block level content had to be indented 4
spaces.  Now the indentation required is determined by the
first line of the list item:  to be included in the list item,
blocks must be indented to the level of the first non-space
content after the list marker. Exception: if are 5 or more spaces
after the list marker, then the content is interpreted as an
indented code block, and continuation paragraphs must be indented
two spaces beyond the end of the list marker.  See the CommonMark
spec for more details and examples.

Documents that adhere to the four-space rule should, in most cases,
be parsed the same way by the new rules.  Here are some examples
of texts that will be parsed differently:

    - a
      - b

will be parsed as a list item with a sublist; under the four-space
rule, it would be a list with two items.

    - a

          code

Here we have an indented code block under the list item, even though it
is only indented six spaces from the margin, because it is four spaces
past the point where a continuation paragraph could begin.  With the
four-space rule, this would be a regular paragraph rather than a code
block.

    - a

            code

Here the code block will start with two spaces, whereas under
the four-space rule, it would start with `code`.  With the four-space
rule, indented code under a list item always must be indented eight
spaces from the margin, while the new rules require only that it
be indented four spaces from the beginning of the first non-space
text after the list marker (here, `a`).

This change was motivated by a slew of bug reports from people
who expected lists to work differently (#3125, #2367, #2575, #2210,
 #1990, #1137, #744, #172, #137, #128) and by the growing prevalance
of CommonMark (now used by GitHub, for example).

Users who want to use the old rules can select the `four_space_rule`
extension.

* Added `four_space_rule` extension.
* Added `Ext_four_space_rule` to `Extensions`.
* `Parsing` now exports `gobbleAtMostSpaces`, and the type
  of `gobbleSpaces` has been changed so that a `ReaderOptions`
  parameter is not needed.

4 files changed, 60 insertions, 14 deletions

diff --git a/test/command/2434.md b/test/command/2434.md
index aa03e5fc3..4f12b6f56 100644
--- a/test/command/2434.md
+++ b/test/command/2434.md
@@ -31,7 +31,7 @@
 
 ```
 % pandoc -t opendocument
-(@)  text
+(@) text
 
     some text
 
diff --git a/test/command/3511.md b/test/command/3511.md
new file mode 100644
index 000000000..b8bcedbb0
--- /dev/null
+++ b/test/command/3511.md
@@ -0,0 +1,46 @@
+```
+% pandoc -t native
+- a
+  - b
+    - c
+
+-     code
+
+1000. one
+
+    not continuation
+^D
+[BulletList
+ [[Plain [Str "a"]
+  ,BulletList
+   [[Plain [Str "b"]
+    ,BulletList
+     [[Plain [Str "c"]]]]]]
+ ,[CodeBlock ("",[],[]) "code"]]
+,OrderedList (1000,Decimal,Period)
+ [[Plain [Str "one"]]]
+,CodeBlock ("",[],[]) "not continuation"]
+```
+
+```
+% pandoc -t native -f markdown+four_space_rule
+- a
+  - b
+    - c
+
+-     not code
+
+1000. one
+
+    continuation
+^D
+[BulletList
+ [[Plain [Str "a"]]
+ ,[Plain [Str "b"]
+  ,BulletList
+   [[Plain [Str "c"]]]]
+ ,[CodeBlock ("",[],[]) "not code"]]
+,OrderedList (1000,Decimal,Period)
+ [[Para [Str "one"]
+  ,Para [Str "continuation"]]]]
+```
diff --git a/test/markdown-reader-more.native b/test/markdown-reader-more.native
index a24417ffe..2e55dbb18 100644
--- a/test/markdown-reader-more.native
+++ b/test/markdown-reader-more.native
@@ -29,13 +29,13 @@ Pandoc (Meta {unMeta = fromList [("author",MetaList [MetaInlines [Str "Author",S
  ,[Plain [Str "three"]]]
 ,Header 2 ("indented-code-at-beginning-of-list",[],[]) [Str "Indented",Space,Str "code",Space,Str "at",Space,Str "beginning",Space,Str "of",Space,Str "list"]
 ,BulletList
- [[CodeBlock ("",[],[]) "code\ncode"]]
-,OrderedList (1,Decimal,Period)
- [[CodeBlock ("",[],[]) "code\ncode"]
- ,[CodeBlock ("",[],[]) "code\ncode"]]
-,BulletList
- [[CodeBlock ("",[],[]) "code\ncode"]
- ,[Plain [Str "no",Space,Str "code"]]]
+ [[CodeBlock ("",[],[]) "code\ncode"
+  ,OrderedList (1,Decimal,Period)
+   [[CodeBlock ("",[],[]) "code\ncode"]
+   ,[CodeBlock ("",[],[]) "code\ncode"]]
+  ,BulletList
+   [[CodeBlock ("",[],[]) "code\ncode"]
+   ,[Plain [Str "no",Space,Str "code"]]]]]
 ,Header 2 ("backslash-newline",[],[]) [Str "Backslash",Space,Str "newline"]
 ,Para [Str "hi",LineBreak,Str "there"]
 ,Header 2 ("code-spans",[],[]) [Str "Code",Space,Str "spans"]
diff --git a/test/markdown-reader-more.txt b/test/markdown-reader-more.txt
index 73c9500a0..0480e41cc 100644
--- a/test/markdown-reader-more.txt
+++ b/test/markdown-reader-more.txt
@@ -84,14 +84,14 @@ $PATH 90 $PATH
 
 ## Indented code at beginning of list
 
--       code
-        code
+-     code
+      code
 
-  1.    code
-        code
+  1.     code
+         code
 
-  12345678.    code
-        code
+  12345678.     code
+                code
 
   -     code
         code