Python string splitlines() removes certain Unicode control characters -


i noticed python's standard string method splitlines() removes crucial unicode control characters well. example

>>> s1 = u'asdf \n fdsa \x1d asdf' >>> s1.splitlines() [u'asdf ', u' fdsa ', u' asdf']

notice how "\x1d" character quietly disappears.

it doesn't happen if string s1 still python bytestring though (without "u" prefix):

>>> s2 = 'asdf \n fdsa \x1d asdf' >>> s2.splitlines() ['asdf ', ' fdsa \x1d asdf']

i can't find information in reference https://docs.python.org/2.7/library/stdtypes.html#str.splitlines.

why happen? other characters "\x1d" (or unichr(29)) affected?

i'm using python 2.7.3 on ubuntu 12.04 lts.

this indeed under-documented; had dig through source code find it.

the unicodetype_db.h file defines linebreaks as:

case 0x000a: case 0x000b: case 0x000c: case 0x000d: case 0x001c: case 0x001d: case 0x001e: case 0x0085: case 0x2028: case 0x2029:

these generated unicode database; codepoint listed in unicode standard line_break property set bk, cr, lf or nl or bidirectional category set b (paragraph break) considered line break.

from unicode data file, version 6 of standard lists u+001d paragraph break:

001d;<control>;cc;0;b;;;;;n;information separator three;;;;

(5th column bidirectional category).

you use regular expression if want limit characters split on:

import re  linebreaks = re.compile(ur'[\n-\r\x85\u2028\u2929]') linebreaks.split(yourtext)

would split text on same set of linebreaks except u+001c, u+001d or u+001e codepoints, 3 data structuring control characters.


Comments

Post a Comment

Popular posts from this blog

javascript - RequestAnimationFrame not working when exiting fullscreen switching space on Safari -

request animation frame stops working when exiting fullscreen clicking on original space's safari window. fine if fullscreen mode canceled escape key or calling cancelfullscreen(). steps reproduce: open https://dl.dropboxusercontent.com/u/769042/prezi/safari-fullscreen.html click "draw raf", kittie appears click "fullscreen", go fullscreen click "draw raf", kittie appears go space original safari was, showing "click exit fullscreen mode", click anywhere, out fullscreen click "draw raf", nothing happens what handling click calling: window.requestanimationframe(draw); which draws on canvas context: function draw() { ctx.drawimage(img, math.random()*500|0, math.random()*400|0, 100, 100); } i checked .hidden , .visibilitystate, updated correctly. tested on osx 10.9.3, safari 7.0.4 (9537.76.4). has workaround/solution other switching old settimeout? this sounds webkit bug 88940 : using reque
Read more

Python ctypes access violation with const pointer arguments -

i have api i'm trying wrap in python (2.7.6 on win7) code using ctypes. here's api: client_dllfunc bool clientapi search_exporttoclipcopy(clienthsearch handle, int channel, lpctstr filename, const time_t& from, const time_t& to, const bool* cameras, int length, bool usepassword, lpctstr password, bool includetextin = false, bool excludeplayer = false); the issue i'm having cameras argument; , i've had issue other const pointer arguments well. here's how i'm wrapping api: def search_exporttoclipcopy(self, hsearch, csearch, filename, tfrom, tto, cameras, length, usepassword, password, includetextin=true, excludeplayer=false): exporttoclipcopy = self.sdkdll.search_exporttoclipcopy exporttoclipcopy.argtypes = [c_int, c_int, c_wchar_p, c_long, c_long, pointer(c_bool), c_int, c_bool, c_wchar_p, c_bool, c_bool] exporttoclipcopy.restype = c_bool exporttoclipcopy(hsearch, csearch, filename, tfrom, tto, cameras, length, usepassword, pass
Read more