WALA Preview - Web Access Log Analyzer

[shawnb61]

Not approved yet, but I figured I'd share with the gang over here.

New mod I've been working on:
https://github.com/sbulen/SMF-Web-Access-Log-Analyzer

Release 1.0.0 download:
https://github.com/sbulen/SMF-Web-Access-Log-Analyzer/releases/download/v1.0.0/WALA-100.zip

I've been helping folks facing bot attacks over at SMF, and have come to realize most folks don't have a way to analyze their web access logs.  E.g., Are these my users???  Which ASNs in Brazil are crawling me?  Do I have users there?

So I wrote a little mod that uses the freebie DBIP database (https://db-ip.com/db/lite.php) to assign country & ASN to both the smf_members table and a web access log.  It has a bunch of canned reports against that.

Toughest challenge was that these are all fairly big files.  So...  I break them all up into small chunks & use the fetch API to get them up there.  Also, a web access log can easily have a few hundred thousand rows of data...  Assigning attributes to all of those rows can take a HUGE amount of time.  I figured out a way to work in small IP ranges, and build lookups in memory instead of using DB joins.   So for the most part, the uploads & attribute assignments just take 2-5 minutes each.  (Via DB lookups they were more like 15-40 minutes each....)

Apache combined log format is required...

[Dave]

Thanks Shawn

[Skhilled]

Wow! Great! I know it was a challenge but I sure hope that you get it approved.  

Now, I'll have to make sure my logs are that format when I get a chance.