----BEGIN CLASS---- [13:58] #startclass [13:58] Roll Call [13:59] Naman Sharma [13:59] Devesh Verma [13:59] Akshay Iyer [13:59] Bhavin Gandhi [13:59] Priyanka saggu [13:59] Shubham Sharma [13:59] Shivam Bansal [13:59] Prashant Sharma [13:59] Akshay Gaikwad [13:59] Neeraj kumar Arya [13:59] Ananyo Maiti [14:00] Gaurav Sitlani [14:00] Manjeet Mehta [14:00] warthog9, Stage is yours :) [14:00] kumar vipin yadav [14:00] Good Morning / Evening, or whatever time of day it may be in your position on the globe :-) [14:00] souvik haldar [14:01] Sandeep Kumar Choudhary [14:01] Sayan Chowdhury [14:01] so I got asked to come and chat with you all on some of my experiences with systems administration, and try and answer any questions you may have to try and stump me with [14:02] Shital Mule [14:02] so I guess some quick background are in order so you've got an idea of my bona fidas, I ran kernel.org for about a decade, and I've done systems administration in some form or capacity continuously since I was in 8th grade [14:03] (needless to say I'm just a bit older than school aged these days ;-) ) [14:04] I've primarily dealt with Linux administration, but I've also done some more basic Windows stuff (which is horribly out of date these days), and Mac as well [14:05] today I have a personal infrastructure of about a dozen or two machines in various colocation facilities and internet connected points around the world [14:05] and I help out with a few open source pieces of infrastructure kinda flung all over the place [14:06] Roll call: sulakhe pooja [14:07] I got into admining my own stuff mostly through a fascination of computers and networking them [14:07] stayed because I prefer being a smidge of a control freak and running my own infrastructure, and helped others because I'm pretty decent at it all told [14:09] I'd probably also bill myself as one of the last "old school unix" style sysadmins, which isn't something you see much of anymore, but to put that phrase into perspective there's just schools of thought from the "old" days that are no longer the watch words they used to be [14:10] "do one thing, do it well, don't do anything else" being one of the old unix sayings [14:11] and it's hard to claim that many tools today aren't getting a bit like this image: https://images-na.ssl-images-amazon.com/images/I/71FHJU17djL._SL1204_.jpg [14:13] so I can probably babble and tell stories the whole hour, but I'm sure there's people with questions that would be more interesting than hearing me try and remember 30 some years of experience [14:14] and if not I'll have to tell a story ;-) [14:15] warthog9, please start with a story :) [14:15] well I'll tell a recent one then [14:16] so there's a surprisingly critical piece of infrastructure in the world that I have control over, can't really say what it is - but lets just say that the world turns because this box exists and does it's job [14:16] this box was running on some hardware (not my personal hardware) that was about 10 years old, and the uptime on the machine was around 2800 days [14:17] we knew it had two failed hard drives, and we were 99.9% sure that if we shut it off, or rebooted it, it wouldn't turn on again. [14:17] Roll call: tabrez khan [14:17] needless to say this is a slightly precarious position for an important box to be in [14:18] so we bought a new box, but if you'll not that 2800 days of uptime means that the last reboot was almost 8 years ago, and the OS landscape in Linux was different back then [14:18] no CentOS/RHEL 6 or 7, 5 was the new thing [14:18] and this was a production box - so it ran RHEL4 [14:19] well in this day and age remembering anything about RHEL4 is a rarity, but turns out after a couple of hours I did remember how it all worked - and nearly live migrated the machine into a VM on the new box [14:20] by hand, and I'd claim uphill both ways in the snow, but I think I was comfortably at home in my office and my office chair so you wouldn't believe that anyway ;-) [14:21] needless to say even getting a copy of the box running in a VM was an accomplishment, but getting the whole works basically live migrated without any appreciable downtime was kinda rewarding [14:21] and in true fashion, no one noticed outside those directly involved [14:21] because everything went well and according to plan [14:22] and that's the dirty secret of infrastructure: no one cares about it till it's broken, then they care a lot ;-) [14:23] so did anyone come up with any questions yet? [14:24] ! [14:24] next [14:24] ! [14:24] I am not able to wrap my head around the live-migration part without any downtime. How was it done? [14:25] there wasn't 0 downtime, but it was minimized [14:25] ! [14:25] trick I pulled is you basically setup a way to sync the old machine to the new machine/vm [14:25] (I love rsync, it's an amazing tool) [14:26] you do have to be careful not to stomp what you need for the new VM to boot/function [14:27] but you sync, sync, sync and basically keep syncing till you have it as close to real-time as you can get, then, disable all the services on the old machine, sync again, disable the ip on the old, enable the ip on the new, and start enabling services [14:27] if you can script it (usually no point) you can probably get the whole transition down to under a minute [14:27] I think our final sync and flip was under 5 minutes [14:28] next [14:28] how much data did you have on that box and any challenges while migrating? [14:28] not a lot of data, less than 100G, and I'd have to log in to double check the sizes [14:29] the main challenege is that a lot of data flows through the machine, constantly, so disruption an downtimes just causes backups and messes later that would take substantially longer to clear out [14:29] and the data that was on disk would change pretty frequently. [14:30] but it was all small files (just lots of them) [14:30] next [14:30] How much knowledge of the kubernetes architecture is required to quickly get into deploying docker containers with it? What according to you is the best way to quickly get started with kubernetes? (deploying on digital ocean on debian) [14:31] so you'll have to take this answer with a bit of a grain of salt, kubernetes and docker are fairly new tech, and they are taking the devopsy world by storm [14:32] and I'm a drag my heals curmudgeon in some respects when it comes to how I think large infrasctures should be built [14:32] (except if you ask my wife, I have a soft spot for experimenting with new file systems she'll point out) [14:33] but overall kubernetes and docker are pretty straight forward, and their general level of documentation out there are quite extensive as everyone gold rushes into working on these [14:34] for the most part though, I haven't found containers to solve many problems in the infrasctures I deal with [14:35] and the biggest problem I have seen them solve is that of people being terrible and lazy packagers and wanting to do slightly naughty things with non-system versioned libraries [14:35] (specifically some centralized management of wireless access points software) [14:35] next [14:36] ! [14:37] next [14:37] How the synchronization between old machine and new machine was working? What was the basic principle behind it? [14:38] just rsync, nothing too fancy [14:38] ! [14:38] if you aren't familiar with rsync it is one of the most amazingly powerful tools for anyone to know [14:38] along with grep, sed, awk, and find [14:38] ! [14:39] ohk warthog9, I am reading. Thank you [14:39] pretty sure you can symplify all systems administration down to combining grep, sed, ask, find, and rsync in odd ways [14:39] but all I was doing was rsyncing from the old machine to the new machine [14:39] something like: [14:40] rsync -avrHSXAx --progress rsync://localhost:3030 /. --exclude=/etc/specificfiles [14:40] and the localhost:3030 thing was an stunnel specifically setup so I didn't have to fight with some other pieces [14:40] but that the connection was still secure [14:40] next [14:41] ohk warthog9, it really helped me to understand the point. Thank you :) [14:42] warthog9, You said the files in the old machine were updated as well with time. So how did you manage to backup the changed version of the synced files? Was that by rsync as well ? [14:42] I mean keeping track of the changed files is a huge task itself [14:43] yup, rsync is a pretty awesome tool, seriously if you haven't looked at it go take a look at it [14:43] if the tl;dr version of rsync is: efficient file tree syncronization with all the extra meta data bits if you need it [14:44] almost like a file level version of dd or dcfldd [14:44] Great. Will have a look at it. [14:44] next [14:44] ! [14:44] Asking questions from specific genre cuz currently stuck with it and you seem to be master of it :) how would you buid strong psql db with k8s? (specific principals to follow for this) [14:44] s/buid/build [14:45] don't know that I would put a database in a k8 like that honestly, as putting databases in VMs or containers and such actually seems like a backwards architecture to me [14:46] DBs and storage you tend to want pretty close to the normal bare metal, fewer layers between it and the performance of the disk [14:47] if I was building a high performance db I'd stick it on it's own box/cluster and carve out chunks of the DB (usernames and dbs) for things to use [14:47] then you also don't take the extra overhead of setup and tear down, but that's my opinion [14:47] next [14:49] ! [14:49] ananyo? [14:49] He must be typing. [14:49] nope [14:50] if he comes back I'll just call him [14:50] next [14:50] warthog9, if you have to suggest 5 setups/examples people here can build using vms at home (to learn), what would be your suggestions? [14:51] well the 0th one is get some virtualization software running somewhere [14:51] kvm/qemu are good choices and free if you are comfortable with Linux [14:51] (ananyo let me answer this question and I'll swing back to yours that you missed) [14:52] warthog9 sure [14:52] if you have a Mac, Windows or Linux - VMware has some good options (full disclosure: I work for VMware) [14:52] but some interesting things to try setting up and playing with (and some of these I have running) [14:53] owncloud / nextcloud (doesn't matter which), squeezebox or another music jukebox kinda server [14:53] do up a windows file sharing setup [14:54] (samba specifically) [14:54] and once your have samba working, figure out how to export the same directories via nfsv4 [14:54] so that's 3 [14:55] 4 go setup some modern website with nginx or apache, and you could even run containers to get it all working (but that could be a bit advanced, but it would be a good learning opportunity) [14:56] and 5 once you have the first 4, go play with collectd and grafana and collect some interesting statistics and graphs from your other VMs ;-) [14:57] you could play with cacti, mrtg, and there's a few other things in that vein too if you wanted to compare / contrast [14:57] but seeing pretty graphs about how your machines are doing is always helpful [14:57] I hope people making a list from the above suggestions :) [14:58] and I think ananyo has fallen off again [14:58] so lets see if there's anytihng in the queue [14:58] next [14:59] Anyone else? [15:01] warthog9, Thank you for amazing story and suggestions. [15:01] you are quite welcome [15:01] ah, ananyo is back. [15:01] ohhh ananyo ask your question quick! [15:01] ananyo, do you want to ask the final question? [15:01] Sorry I am having a very bad connection [15:02] What are the essential skills to be a sysadmin? Do you need to have a solid understanding of the Linux FHS? Also Linux/Unix which type of machines are most used these days as servers ? [15:03] ohhh, good question [15:03] the first skill is reading the documentation for the software, and the second is realizing the documentation is probably terrible and you are going to have to figure it out anyway [15:04] but knowing how your chosen OS works, really well, it a boon [15:04] and it'll help you debug, and understand what's going on when things don't work [15:05] when they work, as I mentioned before, no one cares ;-) [15:05] when they are broke, everyone cares [15:05] sounds like folks need to go take a look at rsync, it's a super good tool if you are syncing or backing up files [15:06] [ba]sh scripting in general is a good thing to know, and knowing a higher level scripting language (perl, python, pick something) is a good idea [15:07] after that be willing to explore on your own, and if you want to take the plunge - running your own infrastructure is a good trial by fire [15:08] 'cause you won't fix what isn't broken *FOR YOU* or that you aren't being yelled at about [15:08] and if your e-mail is down, there's a lot of motivation to fix things ;-) [15:08] (not that I'd recommend to too many people to run mail servers these days because that's gotten hard, just using it as an example) [15:09] so yeah, that's what I'd start with [15:09] there's lots of things to learn [15:10] and lots of systems architecture to argue about the whys and why nots [15:10] warthog9, Thanks for your suggestions [15:10] anyone else now that you've seen that? [15:10] warthog9, Thank you once again. [15:10] Ending the session. ----END CLASS----