java删除(替换)不可见的unicode/utf-8字符(主要是html显示不了的字符)
今天遇到一个问题,由于编辑人员从excel等7788的地方copy内容过来,其中有不可见的字符,导致输出内容看上去是对的,其实是多了一个零长度的字符(比如:0000200B ZERO WIDTH SPACE),下面的代码基本解决了以上问题。
?
?
<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8" /><script>window.onload=function(){var zero_len_charcode=[]var div = document.getElementById('ttt')for(var i=0;i<=0xffff;i++){//var c = String.fromCharCode(i);if (!c) continue;div.innerHTML = "|"+c+"|";if (div.offsetWidth == 16) //这里的16是我机器的2个'||'的显示长度,请自己调节zero_len_charcode.push(i.toString(16));}console.log('0x'+zero_len_charcode.join(",0x"))}</script></head><body><span id=ttt></span></body></html>
?上面的js代码用于提取出看不见的字符(主要是在html上看不见,也就是前面说的0长度的),copy上面console输出的内容,然后去java里做服务器端处理(为什么这么麻烦,因为swing我不熟悉,而且也不一定可以做这样的长度适应判断)
?
private static String zerolize(String s) {if (s.length() < 4) {s = "000".substring(0, 4 - s.length()) + s;}return s;}public static void main(String[] args) throws IOException {int[] input = new int[] { 0x7f, 0x80, 0x81, 0x82, 0x83, 0x84, 0x85, 0x86, 0x87, 0x88, 0x89, 0x8a,0x8b, 0x8c, 0x8d, 0x8e, 0x8f, 0x90, 0x91, 0x92, 0x93, 0x94, 0x95, 0x96, 0x97, 0x98, 0x99,0x9a, 0x9b, 0x9c, 0x9d, 0x9e, 0x9f, 0xad, 0x483, 0x484, 0x485, 0x486, 0x487, 0x488, 0x489,0x559, 0x55a, 0x58a, 0x591, 0x592, 0x593, 0x594, 0x595, 0x596, 0x597, 0x598, 0x599, 0x59a,0x59b, 0x59c, 0x59d, 0x59e, 0x59f, 0x5a0, 0x5a1, 0x5a2, 0x5a3, 0x5a4, 0x5a5, 0x5a6, 0x5a7,0x5a8, 0x5a9, 0x5aa, 0x5ab, 0x5ac, 0x5ad, 0x5ae, 0x5af, 0x5b0, 0x5b1, 0x5b2, 0x5b3, 0x5b4,0x5b5, 0x5b6, 0x5b7, 0x5b8, 0x5b9, 0x5ba, 0x5bb, 0x5bc, 0x5bd, 0x5bf, 0x5c1, 0x5c2, 0x5c4,0x5c5, 0x5c6, 0x5c7, 0x606, 0x607, 0x608, 0x609, 0x60a, 0x63b, 0x63c, 0x63d, 0x63e, 0x63f,0x674, 0x6e5, 0x6e6, 0x70f, 0x76e, 0x76f, 0x770, 0x771, 0x772, 0x773, 0x774, 0x775, 0x776,0x777, 0x778, 0x779, 0x77a, 0x77b, 0x77c, 0x77d, 0x77e, 0x77f, 0xa51, 0xa75, 0xb44, 0xb62,0xb63, 0xc62, 0xc63, 0xce2, 0xce3, 0xd62, 0xd63, 0x135f, 0x200b, 0x200c, 0x200d, 0x200e,0x200f, 0x2028, 0x2029, 0x202a, 0x202b, 0x202c, 0x202d, 0x202e, 0x2044, 0x2071, 0xf701,0xf702, 0xf703, 0xf704, 0xf705, 0xf706, 0xf707, 0xf708, 0xf709, 0xf70a, 0xf70b, 0xf70c,0xf70d, 0xf70e, 0xf710, 0xf711, 0xf712, 0xf713, 0xf714, 0xf715, 0xf716, 0xf717, 0xf718,0xf719, 0xf71a, 0xfb1e, 0xfc5e, 0xfc5f, 0xfc60, 0xfc61, 0xfc62, 0xfeff, 0xfffc };StringBuilder b = new StringBuilder();int lastContinuous = -1;int span = 0;for (int i = 0; i < input.length; i++) {if (lastContinuous == -1 && i < input.length - 1 && input[i] + 1 == input[i + 1]) {lastContinuous = input[i];span = 1;} else {if (input[i] == lastContinuous + span) {span++;} else if (lastContinuous != -1) {if (b.length() > 0)b.append("|");b.append(String.format("[\\u%s-\\u%s]", zerolize(Integer.toHexString(lastContinuous)),zerolize(Integer.toHexString(lastContinuous + span - 1))));span = 0;lastContinuous = -1;i--;} else {b.append("|\\u" + zerolize(Integer.toHexString(input[i])));}}}if (lastContinuous != -1) {if (b.length() > 0)b.append("|");b.append(String.format("[\\u%s-\\u%s]", zerolize(Integer.toHexString(lastContinuous)),zerolize(Integer.toHexString(lastContinuous + span - 1))));}System.out.println(b.toString());Pattern pattern = Pattern.compile(b.toString());Matcher matcher = pattern.matcher("helloworld\ufffc撒范德萨分的阿萨德");System.out.println(matcher.replaceAll(""));}?
最后组成的正则如下:
[\u007f-\u009f]|\u00ad|[\u0483-\u0489]|[\u0559-\u055a]|\u058a|[\u0591-\u05bd]|\u05bf|[\u05c1-\u05c2]|[\u05c4-\u05c7]|[\u0606-\u060a]|[\u063b-\u063f]|\u0674|[\u06e5-\u06e6]|\u070f|[\u076e-\u077f]|\u0a51|\u0a75|\u0b44|[\u0b62-\u0b63]|[\u0c62-\u0c63]|[\u0ce2-\u0ce3]|[\u0d62-\u0d63]|\u135f|[\u200b-\u200f]|[\u2028-\u202e]|\u2044|\u2071|[\uf701-\uf70e]|[\uf710-\uf71a]|\ufb1e|[\ufc5e-\ufc62]|\ufeff|\ufffc