panettone: some unicode codepoints are converted into gibberish
#107
Opened by sterni at
I tried to use the set union operator U+222A in b/104, but it went horribly wrong, as you can see.
Pasting it here again: ∪
.
interestingly, the email came through perfectly fine.
grfn at 2021-04-06T02·01+00
Seems to be caused by
RENDER-MARKDOWN
. Issue titles are completely fine for example, but I can reproduce the issue on the repl:* (in-package :panettone) #<PACKAGE "PANETTONE"> * (render-markdown "⊂∫∪") "<p>ââ«âª</p> "
sterni at 2021-04-06T08·38+00
Turns out that the problem is caused by
drakma
or more specificallyflexi-stream
which changesread-char
in a way that it chokes on UTF-8 (even thoughexternal-format
isutf-8
):(let ((s (drakma:http-request "http://localhost:4238/markdown" :method :post :content-type "application/json" :accept "application/json" :content "{ \"markdown\": \"∪nion\" }" :external-format-out :utf-8 :external-format-in :utf-8 :want-stream t))) (loop for x = (read-char s nil nil) until (not x) collect x))
Results in:
(#\{ #\" #\m #\a #\r #\k #\d #\o #\w #\n #\" #\: #\" #\< #\p #\> #\LATIN_SMALL_LETTER_A_WITH_CIRCUMFLEX #\Character-Tabulation-Set #\FEMININE_ORDINAL_INDICATOR #\n #\i #\o #\n #\< #\/ #\p #\> #\\ #\n #\" #\})
sterni at 2021-04-06T11·32+00
- closed this issue at 2021-04-09T19·12+00