ServerEngine Integration & Windows support Ritta Narita Fluentd Meetup 2016 About me. Ritta Narita joined Treasure Data after internship in 2015 2 @naritta @narittan Fluentd v0.14 ServerEngine is integrated What’s ServerEngine? A framework to implement robust multiprocess servers like Unicorn worker worker worker super visor Server Dynamic reconfig live restart Heartbeat via pipe auto restart multi process/thread how to use ServerEngine workersuper visor Server module MyServer def before_run @sock = TCPServer.new( config[:bind], config[:port]) end attr_reader :sock end module MyWorker def run c = server.sock.accept c.write "Awesome work!" end def stop c.close end end 1. write server module and worker module how to use ServerEngine workersuper visor Server se = ServerEngine.create(MyServer, MyWorker, { daemonize: true, log: 'myserver.log', pid_path: 'myserver.pid', worker_type: 'process', workers: 4, bind: '0.0.0.0', port: 9071, }) se.run 2. write configuration and run module MyWorker def spawn(process_manager) process_manager.spawn(env, config[:spawn_cmd]) end end ServerEngine.create(MyServer, MyWorker, { worker_type: ‘spawn’, spawn_cmd: cmd }) ServerEngine.create(MyServer, MyWorker, { worker_type: ‘thread' }) workerServer workerServer workerServer spawn thread processServerEngine.create(MyServer, MyWorker, { worker_type: ‘process' }) worker type What’s the advantages for integrating serverengine into fluentd? serverengine advantages Auto restart live restart socket manager signal handler log rotation Auto restart live restart socket manager signal handler log rotation serverengine advantages if worker died unexpectedly… worker worker super visor Server worker if worker died unexpectedly… worker worker worker super visor Server if worker died unexpectedly… worker worker super visor Server It’s detected by heartbeat and server launch worker again. worker Auto restart live restart socket manager signal handler log rotation serverengine advantages worker super visor Server tcp tcp send SIGHUP live restart super visor Server worker tcp kill the worker, leaving server’s condition live restart worker super visor Server worker tcp start new worker and read config again live restart worker super visor Server worker tcp tcp share the TCP with living zero downtime live restart Signal and RPC server api/config.reloadlive restart/config reload HUP /api/processes .interruptWorkers INT /api/processes .killWorkers /api/plugins. flushBuffers system config reload USR2 TERM USR1flush buffers shutdown worker super visor Server you can reload system config without restart system config reload send USR2 check mtime and previous load time, if there is new one, reload! serverengine config cash new config Auto restart live restart socket manager signal handler log rotation serverengine advantages Socket Manager worker Socket Manager client Socket Manager server Server 26 ①socket listen request②find or create listening socket ③share the socket FD(unix)/ duplicated socket info(windows) UDS(unix)/ TCP(windows) ④get socket and accept in worker how to use Socket Manager worker Socket Manager client Socket Manager server Server 27 #generate path for UDS/TCP for server connection socket_manager_path = ServerEngine::SocketManager::Server.generate_path #open socket manager server socket_manager_server = ServerEngine::SocketManager::Server.open(socket_manager_path) In server worker Socket Manager client Socket Manager server Server 28 #get socket manager client from socket manager path socket_manager = ServerEngine::SocketManager::Client.new(socket_manager_path) #get listening socket from socket manager lsock = socket_manager.listen_tcp(bind, port) In worker how to use Socket Manager What’s the merit of Socket Manager? worker worker worker Socket Manager client Socket Manager client Socket Manager client Socket Manager server super visor Server 29 we can use multicore easily in future easy to use multicore power fully 30 with SocketManagerwith in_multiprocess plugin type multiprocess cmdline -c /etc/td-agent/td-agent-child1.conf cmdline -c /etc/td-agent/td-agent-child2.conf #/etc/td-agent/td-agent-child1.conf type forward port 24224 #/etc/td-agent/td-agent-child2.conf type forward port 24225 type forward port 24224 setting when using 2 core Auto restart live restart socket manager signal handler log rotation serverengine advantages Queue based signal handling When user sent USR1 and QUIT… signal handler thread user USR1 send USR1 Queue based signal handling When user sent USR1 and QUIT… signal handler thread user USR1 USR1 Queue based signal handling When user sent USR1 and QUIT… signal handler thread user QUIT USR1 send QUIT Queue based signal handling When user sent USR1 and QUIT… signal handler thread user USR1 QUIT Queue based signal handling When user sent USR1 and QUIT… signal handler thread user USR1 QUIT run! Queue based signal handling When user sent USR1 and QUIT… signal handler thread user QUIT Queue based signal handling When user sent USR1 and QUIT… signal handler thread user QUIT run! Auto restart live restart socket manager signal handler log rotation serverengine advantages support new features (These are already ported to ruby core.) ・multiprocess aware log rotation ・reopening of log file ・'trace' level, which is lower level than ‘debug' ServerEngine.create(MyServer, MyWorker, { log: 'myserver.log', log_level: 'debug', log_rotate_age: 5, log_rotate_size: 1*1024*1024, }) summary windows ・signal → rpc handler ・no socket FD and UDS ・fork based → spawn based ・live & auto restart ・multicore with socket manager ・sophisticated signal handler ・easier log rotation Thank you! appendix unix→fork based damon windows→spawn based damon damonize fluentd server engine ①set windows_daemon_cmdline in configuration ②run windows_daemon_cmdline when damonize ③windows_daemon_cmdline runs daemon.rb ④run_server with servermodule and workermodule
Comments
Report "Fluentd Meetup 2016 - ServerEngine Integration & Windows support"