DEGREE EARNED: 08/11/2006 BACHELOR OF ARTS
Well it's ABOUT FREAKIN' TIME!
DSL came on Wednesday, so we're finally back in touch with the rest of the world. AAAnd, my transcripts FINALLY went through like, TWO DAYS AGO, so now I've actually officially graduated! Now I just have to wait for the degree to get mailed. Took freakin' long enough, sheesh...
Not much been going on here.. Debated for a while yesterday whether to get in line for a PS3, but after considering a) the cost, b) the games that'd be available, I decided to just forget about it & reconsider if/when a new Ratchet & Clank came out for it (been replaying those again lately). Although it's quite possible that by that time, it'll still be impossible to get one. Apparently there were still lines at around noon today (Girl passed a Bic on the way to a lesson), but getting in line at that point would've been pointless. There were signs up at Yodobashi last night saying not to line up overnight, but there were still a few guys (and Chinese women) hanging around that you could tell were going to start lining up as soon as they closed & went away... Apparently Girl passed a huge group of Chinese people loading Bic bags into the back of a van, presumably to bring back to their unlicensed Akiba shop or in-house ebay store... -_-; oh well. I briefly debated getting in line & buying one just to sell on ebay myself, after hearing shit about dumbasses paying $3000 just for a freakin' pre-order registration slip (which means they'd have to pay for the actual unit on top of that), but eh.. fuckit. Even though the people willing to pay that much for that kinda shit deserve it, it still seems like an asshole thing to do. oh well.
Need to get my pictures updated.. took a few from the office the other day, since we've been able to see Mt. Fuji from the windows, which is damn nice looking at sunset... you can see the odaiba ferris wheel from the other corner. Too bad visibility is so crappy most of the time that you can't see much during the day.. don't know if it's clouds or smog or what, but it's l4m3. Then again, the blinds are closed most of the day anyway 'cause of the glare, but whatever.
Work's been mostly transitioning back from Unix (which I had to take over when the last Unix guy left a couple weeks before the new guy was supposed to start) to Sybase (which I was originally hired for), and rewriting all the account management tools, partly since they just plain suck, partly 'cause there's 4 separate scripts for every single application (create, delete, password reset, modify), and partly 'cause the stored procedures are all changing, so they have to be rewritten anyway. Managed to write one big-ass tool to handle everything, w/ all the application info in a giant "config file" (i.e. raw hash of hashes (of hashes of arrays of hashes etc.) declaration). But that was nothing...
There's this monthly report that we run against a log that we use to keep track of what we do (which is in a freakin' ACCESS database, which is the next thing I wanna rewrite, but anyway)... the report basically grabs the employee ID that we enter into it, & grabs the user's department info & crap from the employee database, which is a gigantic (450k+ line) csv file. It takes about 4-5 hours to run the damn report, and here's why:
1) export the list of IDs, & run script A on it
2) script A loads the list, then the employee db, then for each line in the list, greps the employee db to see if there's any matches. If there's not, it writes "not found". If there are, then it greps the file AGAIN to get the data.
3) take the "not found" stuff from step 2, and run a script that checks for those users in a database (since the db just sets 'active' to 0 for employees who've left the company- the csv file removes them and only includes active employees). If it finds it in the db, output the employee info. If not found again, put 'not found'.
4) Take the "found" stuff and APPEND IT TO THE EMPLOYEE DB, and then run step 2 AGAIN.
5) After all that's done, stick it back in the excel report
6) Grab ANOTHER column from the report (department code), and export it to another file
7) run another report that grabs the name & location for the department code (by grepping it from a ~14k-line file
Now, doing a 'grep' from the command line on the employee DB takes literally 10 seconds. Let's say that running it in perl is faster, since it's all in memory, and knock it down to 2.5 seconds. But even still- grep has to compare EVERY SINGLE RECORD, even after it finds a match, since it's designed to find ALL matches, not just the first one. And since the script in step 2 greps the file TWICE for each record in the report (about 1500), and the report is run TWICE (again after finding the stuff that wasn't found the first time), you're bascially guaranteeing 10s * the number of rows, aka over 4 hours for 1500 rows. On top of that, the database script in step 3 CONNECTS AND DISCONNECTS TO THE DATABASE ON EVERY LOOKUP. That's like, the most time & resource-intensive step, and EVERY man page on database shit will tell you to open the connection ONCE and re-use it.
So, I rewrote it all into one script w/ the following logic:
1) load the employee db into a HASH w/ the employee ID as the key (hash lookups are pretty much instantaneous).
2) loop through the file. If the hash key exists, output the data. if it doesn't, look it up in the (already opened) database. If it's still not there, put 'not found'. Look up the department info at the same time
The script now takes 2.5 MINUTES (about 2m until it says "Done!" (which means the script is finished)- the last 30s or so is freeing the memory, I assume). Sheesh. That's literally an improvement of 100x, or 10000%. For fuck's sake, if the guy who wrote that script still worked there I woulda slapped him just for the sheer inefficiency of the thing.
Anyway... Not much else going on here. Work, TV, games. Did karaoke for a couple hours last weekend, but night + weekend + 1 drink each = nearly $50 for the 2 of us. Will have to go back to the daytime thing (11-8pm = ¥980). Still need to find new karaoke buddies... oh well.
That's about it for now.. I'll update once I get all my pix uploaded- there's a bunch of the new place & stuff...
DSL came on Wednesday, so we're finally back in touch with the rest of the world. AAAnd, my transcripts FINALLY went through like, TWO DAYS AGO, so now I've actually officially graduated! Now I just have to wait for the degree to get mailed. Took freakin' long enough, sheesh...
Not much been going on here.. Debated for a while yesterday whether to get in line for a PS3, but after considering a) the cost, b) the games that'd be available, I decided to just forget about it & reconsider if/when a new Ratchet & Clank came out for it (been replaying those again lately). Although it's quite possible that by that time, it'll still be impossible to get one. Apparently there were still lines at around noon today (Girl passed a Bic on the way to a lesson), but getting in line at that point would've been pointless. There were signs up at Yodobashi last night saying not to line up overnight, but there were still a few guys (and Chinese women) hanging around that you could tell were going to start lining up as soon as they closed & went away... Apparently Girl passed a huge group of Chinese people loading Bic bags into the back of a van, presumably to bring back to their unlicensed Akiba shop or in-house ebay store... -_-; oh well. I briefly debated getting in line & buying one just to sell on ebay myself, after hearing shit about dumbasses paying $3000 just for a freakin' pre-order registration slip (which means they'd have to pay for the actual unit on top of that), but eh.. fuckit. Even though the people willing to pay that much for that kinda shit deserve it, it still seems like an asshole thing to do. oh well.
Need to get my pictures updated.. took a few from the office the other day, since we've been able to see Mt. Fuji from the windows, which is damn nice looking at sunset... you can see the odaiba ferris wheel from the other corner. Too bad visibility is so crappy most of the time that you can't see much during the day.. don't know if it's clouds or smog or what, but it's l4m3. Then again, the blinds are closed most of the day anyway 'cause of the glare, but whatever.
Work's been mostly transitioning back from Unix (which I had to take over when the last Unix guy left a couple weeks before the new guy was supposed to start) to Sybase (which I was originally hired for), and rewriting all the account management tools, partly since they just plain suck, partly 'cause there's 4 separate scripts for every single application (create, delete, password reset, modify), and partly 'cause the stored procedures are all changing, so they have to be rewritten anyway. Managed to write one big-ass tool to handle everything, w/ all the application info in a giant "config file" (i.e. raw hash of hashes (of hashes of arrays of hashes etc.) declaration). But that was nothing...
There's this monthly report that we run against a log that we use to keep track of what we do (which is in a freakin' ACCESS database, which is the next thing I wanna rewrite, but anyway)... the report basically grabs the employee ID that we enter into it, & grabs the user's department info & crap from the employee database, which is a gigantic (450k+ line) csv file. It takes about 4-5 hours to run the damn report, and here's why:
1) export the list of IDs, & run script A on it
2) script A loads the list, then the employee db, then for each line in the list, greps the employee db to see if there's any matches. If there's not, it writes "not found". If there are, then it greps the file AGAIN to get the data.
3) take the "not found" stuff from step 2, and run a script that checks for those users in a database (since the db just sets 'active' to 0 for employees who've left the company- the csv file removes them and only includes active employees). If it finds it in the db, output the employee info. If not found again, put 'not found'.
4) Take the "found" stuff and APPEND IT TO THE EMPLOYEE DB, and then run step 2 AGAIN.
5) After all that's done, stick it back in the excel report
6) Grab ANOTHER column from the report (department code), and export it to another file
7) run another report that grabs the name & location for the department code (by grepping it from a ~14k-line file
Now, doing a 'grep' from the command line on the employee DB takes literally 10 seconds. Let's say that running it in perl is faster, since it's all in memory, and knock it down to 2.5 seconds. But even still- grep has to compare EVERY SINGLE RECORD, even after it finds a match, since it's designed to find ALL matches, not just the first one. And since the script in step 2 greps the file TWICE for each record in the report (about 1500), and the report is run TWICE (again after finding the stuff that wasn't found the first time), you're bascially guaranteeing 10s * the number of rows, aka over 4 hours for 1500 rows. On top of that, the database script in step 3 CONNECTS AND DISCONNECTS TO THE DATABASE ON EVERY LOOKUP. That's like, the most time & resource-intensive step, and EVERY man page on database shit will tell you to open the connection ONCE and re-use it.
So, I rewrote it all into one script w/ the following logic:
1) load the employee db into a HASH w/ the employee ID as the key (hash lookups are pretty much instantaneous).
2) loop through the file. If the hash key exists, output the data. if it doesn't, look it up in the (already opened) database. If it's still not there, put 'not found'. Look up the department info at the same time
The script now takes 2.5 MINUTES (about 2m until it says "Done!" (which means the script is finished)- the last 30s or so is freeing the memory, I assume). Sheesh. That's literally an improvement of 100x, or 10000%. For fuck's sake, if the guy who wrote that script still worked there I woulda slapped him just for the sheer inefficiency of the thing.
Anyway... Not much else going on here. Work, TV, games. Did karaoke for a couple hours last weekend, but night + weekend + 1 drink each = nearly $50 for the 2 of us. Will have to go back to the daytime thing (11-8pm = ¥980). Still need to find new karaoke buddies... oh well.
That's about it for now.. I'll update once I get all my pix uploaded- there's a bunch of the new place & stuff...