云计算实战 (海量日志管理)hadoop + scribe -- scribe配置详解
port: assigned to variable “port”
max_msg_per_second:
max_queue_size: in bytes
check_interval: in seconds
new_thread_per_category: yes/no
num_thrift_server_threads:
Example:
port=1463max_msg_per_second=2000000max_queue_size=10000000check_interval=3
Store ConfigurationScribe Server determines how to log messages based on the Stores defined in the configuration. Every store must specify what message category it handles with three exceptions:
default store: The ‘default’ category handles any category that is not handled by any other store. There can only be one default store.
prefix stores: If the specified category ends in a *, the store will handle all categories that begin with the specified prefix.
multiple categories: Can use ‘categories=’ to create multiple stores with a single store definition.
In the above three cases, Scribe will create a subdirectory for each unique category in File Stores (unless new_thread_per_category is set to false).
Store Configuration Variablescategory: Determines which messages are handled by this store
type:
target_write_size: 16,384 bytes by default
max_batch_size: 1,024,000 bytes by default (may not be in open-source yet)
max_write_interval: 10 seconds by default
must_succeed: yes/no
Example:
<store>category=statisticstype=filetarget_write_size=20480max_write_interval=2</store>
File Store ConfigurationFile Stores write messages to a file.
file_path: defaults to “/tmp”
base_filename: defaults to category name
use_hostname_sub_directory: yes/no, default no
sub_directory: string
rotate_period: “hourly”, “daily”, “never”, or number[suffix]; “never” by default
rotate_hour: 0-23, 1 by default
rotate_minute?0-59, 15 by default
max_size: 1,000,000,000 bytes by default
write_meta: “yes” or anything else; false by default
fs_type: supports two types “std” and “hdfs”. “std” by default
chunk_size: 0 by default. If a chunk size is specified, no messages within the file will cross chunk boundaries unless there are messages larger than the chunk size
add_newlines: 0 or 1, 0 by default
create_symlink: “yes” or anything else; “yes” by default
write_stats: yes/no, yes by default
max_write_size: 1000000 bytes by default. The file store will try to flush the data out to the file system in chunks of max_write_size of bytes. max_write_size cannot be more than max_size. Say due to target_write_size a certain number of messages were buffered. And then the file store was called to save these messages. The file-store will save these messages at least max_write_size bytes sized chunks at a time. The last write that the file store will make can be smaller than max_write_size.
Example:
<store>category=sprocketstype=filefile_path=/tmp/sprocketsbase_filename=sprockets_logmax_size=1000000add_newlines=1rotate_period=dailyrotate_hour=0rotate_minute=10max_write_size=4096</store>
Network Store ConfigurationNetwork Stores forward messages to other Scribe Servers. Scribe keeps persistent connections open as long as it is able to send messages. (It will only re-open a connection on error or if the downstream machine is overloaded). Scribe will send messages in batches during normal operation based on how many messages are currently sitting in the queue waiting to be sent. (If Scribe is backed up and buffering messages to local disk, Scribe will send messages in chunks based on the buffer file sizes.)
remote_host: name or ip of remote host to forward messages
remote_port: port number on remote host
timeout: socket timeout, in MS; defaults to DEFAULT_SOCKET_TIMEOUT_MS, which is set to 5000 in store.h
use_conn_pool: “yes” or anything else; defaults to false
Example:
<store>category=defaulttype=networkremote_host=halremote_port=1465</store>
Buffer Store ConfigurationBuffer Stores must have two sub-stores named “primary” and “secondary”. Buffer Stores will first attempt to Log messages to the primary store and only log to the secondary if the primary is not available. Once the primary store comes back online, a Buffer store will read messages out of the secondary store and send them to the primary store (unless replay_buffer=no). Only stores that are readable (store that implement the readOldest() method) may be used as secondary store. Currently, the only readable stores are File Stores and Null Stores.
max_queue_length: 2,000,000 messages by default
buffer_send_rate: 1 by default
retry_interval: 300 seconds by default
retry_interval_range: 60 seconds by default
replay_buffer: yes/no, default yes
Example:
<store>category=defaulttype=bufferbuffer_send_rate=1retry_interval=30retry_interval_range=10 <primary> type=network remote_host=wopr remote_port=1456 </primary> <secondary> type=file file_path=/tmp base_filename=thisisoverwritten max_size=10000000 </secondary></store>
Bucket Store ConfigurationBucket Stores will hash messages to multiple files using a prefix of each message as the key.
You can define each bucket implicitly(using a single ‘bucket’ definition) or explicitly (using a bucket definition for every bucket). Bucket Stores that are defined implicitly must have a substore named “bucket” that is either a File Store, Network store or ThriftFile Store (see examples).
num_buckets: defaults to 1
bucket_type: “key_hash”, “key_modulo”, or “random”
delimiter: must be an ascii code between 1 and 255; otherwise the default delimiter is ‘:’
remove_key: yes/no, defaults to no
bucket_subdir: the name of each subdirectory will be this name followed by the bucket number if a single ‘bucket’ definition is used
Example:
<store>category=bucket_metype=bucketnum_buckets=5bucket_subdir=bucketbucket_type=key_hashdelimiter=58 <bucket> type=file fs_type=std file_path=/tmp/scribetest base_filename=bucket_me </bucket></store>
Instead of using a single ‘bucket’ definition for all buckets, you can specify each bucket explicitly:
<store>category=bucket_metype=bucketnum_buckets=2bucket_type=key_hash <bucket0> type=file fs_type=std file_path=/tmp/scribetest/bucket0 base_filename=bucket0 </bucket0> <bucket1> ... </bucket1> <bucket2> ... </bucket2></store>
You can also bucket into network stores as well:
<store>category=bucket_metype=bucketnum_buckets=2bucket_type=random <bucket0> type=file fs_type=std file_path=/tmp/scribetest/bucket0 base_filename=bucket0 </bucket0> <bucket1> type=network remote_host=wopr remote_port=1463 </bucket1> <bucket2> type=network remote_host=hal remote_port=1463 </bucket2></store>
Null Store ConfigurationNull Stores can be used to tell Scribe to ignore all messages of a given category.
(no configuration parameters)
Example:
<store>category=tps_report*type=null</store>
Multi Store ConfigurationA Multi Store is a store that will forward all messages to multiple sub-stores.
A Multi Store may have any number of substores named “store0”, “store1”, “store2”, etc
report_success: “all” or “any”, defaults to “all”
Example:
<store>category=defaulttype=multitarget_write_size=20480max_write_interval=1 <store0> type=file file_path=/tmp/store0 </store0> <store1> type=file file_path=/tmp/store1 </store1></store>
Thriftfile Store ConfigurationA Thriftfile store is similar to a File store except that it stores messages in a Thrift TFileTransport file.
file_path: defaults to “/tmp”
base_filename: defaults to category name
rotate_period: “hourly”, “daily”, “never”, or number[suffix]; “never” by default
rotate_hour: 0-23, 1 by default
rotate_minute?0-59, 15 by default
max_size: 1,000,000,000 bytes by default
fs_type: currently only “std” is supported; “std” by default
chunk_size: 0 by default
create_symlink: “yes” or anything else; “yes” by default
flush_frequency_ms: milliseconds, will use TFileTransport default of 3000ms if not specified
msg_buffer_size: in bytes, will use TFileTransport default of 0 if not specified
Example:
<store>category=sprocketstype=thriftfilefile_path=/tmp/sprocketsbase_filename=sprockets_logmax_size=1000000flush_frequency_ms=2000</store>
Last edited by?zshao,?September 13, 2010?
1 楼 wanweiaiaqiang 2011-07-04 正找这个,要是能够翻译成中文就更好了,中英文对照!