ERROR 1135 (HY000): Can't create a new thread (errno 11); 引发的血案
睡梦中,被红木城的电话call起来了,说application页面报500错误了。
打开电脑,登录公司内部oc通讯工具,leader以及manager以及sa以及minotor都在线呢,大家正在讨论中,他们已经可以确认问题在db了,我赶紧去后台检查db。
马上登录数据库,用root检查db状况:
[xxx@xxx-db003db1 mysqldata]$ mysql -uroot
ERROR 1135 (HY000): Can't create a new thread (errno 11); if you are not out of available memory, you can consult the manual for a possible OS-dependent bug
看到这个问题, wow,比较诡异啊,can't create a new thread,根据以往的经验:
1 磁盘满了
2 client connection满了。
马上执行
[xxx@xxx-db003db1 mysqldata]$ df -h
/dev/mapper/mysqldata
230G 230G 0G 100% /mysqldata
郁闷啊,怎么会100%呢,去/mysqldata目录看,N多binlog啊,于是OC通讯组里面汇报情况后,找sa去物理机上面给/mysqldata目录多分配100M的空间(这个步骤很重要,不然无法登陆MySQL命令行界面),然后马上去清理过多的bilog。
先去从库执行 show slave status\G找到最后的position
Master_Log_File: mysql-bin.001992
然后在出问题的主库上面登陆MySQL界面执行clear操作:
mysql> purge master logs to 'mysql-bin.012082';
Query OK, 0 rows affected (21.21 sec)
mysql> exit
Bye
[xxx@xxx-db003db1 mysqldata]$ df -h
/dev/mapper/mysqldata
230G 115G 0G 50% /mysqldata
OK,磁盘请空了。
然后login check下,能正常登录了。
[xxx@xxx-db003db1 mysqldata]$ mysql -uroot
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 15697279
Server version: 5.5.25a-log MySQL Release
Copyright (c) 2000, 2011, Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql>
做完了,在OC通讯组里面跟大家汇报下,然后发email出去。今天休息,看来明天要好好查下,为什么100%了才报警。