D语言,WEB 日志正则分解,管道处理进入数据库
$ cat match.d import std.regex;import std.stdio;import std.string;import std.array;void main(){auto r = regex(`^(\S+) (\S+) (\S+) \[(.+)\] "([^"]+)" ([0-9]{3}) ([0-9]+) "([^"]+)" "([^"]+)" "([^"]+)"`);foreach(line; stdin.byLine){ foreach(m; match(line, r)){auto c = m.captures;c.popFront();auto value = join(c, "","");auto sql = format("insert into log(remote_addr,unknow,remote_user,time_local,request,status,body_bytes_sent,http_referer,http_user_agent,http_x_forwarded_for) value("%s");", value );writeln(sql);}}}
?编译
$ dmd match.d 编译优化(减肥)$ strip match
测试
$ cat access.log | ./match输出类似insert into log(remote_addr,unknow,remote_user,time_local,request,status,body_bytes_sent,http_referer,http_user_agent,http_x_forwarded_for) value("192.168.2.76","-","-","19/Mar/2013:15:24:20 +0800","GET /favicon.ico HTTP/1.1","404","193","-","Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.11 (KHTML, like Gecko) Chrome/17.0.963.84 Safari/535.11 SE 2.X MetaSr 1.0","-");
??
用法
?
$ cat access.log | match | mysql -hlocalhost -ulog -p123456 logging
?
高级用法, 实时处理日志
上面程序稍加修改即可实现Hbase,?Hypertable?插入
?
?
?
?
?
?
'$status $body_bytes_sent "$http_referer" '