panettone: some unicode codepoints are converted into gibberish

#107
Opened by sterni at 2021-04-05T00·16+00

I tried to use the set union operator U+222A in b/104, but it went horribly wrong, as you can see.

Pasting it here again: ∪.

  1. interestingly, the email came through perfectly fine.

    grfn at 2021-04-06T02·01+00

  2. Seems to be caused by RENDER-MARKDOWN. Issue titles are completely fine for example, but I can reproduce the issue on the repl:

    * (in-package :panettone)
#<PACKAGE "PANETTONE">
* (render-markdown "⊂∫∪")
"<p>ââ«âª</p>
"

    sterni at 2021-04-06T08·38+00

  3. Turns out that the problem is caused by drakma or more specifically flexi-stream which changes read-char in a way that it chokes on UTF-8 (even though external-format is utf-8):

    (let
  ((s (drakma:http-request
       "http://localhost:4238/markdown"
       :method :post :content-type "application/json"
       :accept "application/json"
       :content "{ \"markdown\": \"∪nion\" }"
       :external-format-out :utf-8 :external-format-in :utf-8
       :want-stream t)))
  (loop for x = (read-char s nil nil) until (not x) collect x))

    Results in:

    (#\{ #\" #\m #\a #\r #\k #\d #\o #\w #\n #\" #\: #\" #\< #\p #\>
 #\LATIN_SMALL_LETTER_A_WITH_CIRCUMFLEX #\Character-Tabulation-Set
 #\FEMININE_ORDINAL_INDICATOR #\n #\i #\o #\n #\< #\/ #\p #\> #\\ #\n #\" #\})

    sterni at 2021-04-06T11·32+00

  4. closed this issue at 2021-04-09T19·12+00