Editing Data Collection

This section outlines how data is collected for the TAKP database and how people can contribute to the project.  Most data collecting does not require much technical ability, so just about anyone with an eye for detail can contribute.

== Data Sources ==

Data for TAKP comes from several primary sources: Log files (from AK or old EQ), AK ShowEQ collects and logs, Torven's NPC stats spreadsheet, Torven's video data, Magelo, Allakhazam, and Live EQ.  Long-time AK players are also an invaluable source of knowledge as they were stuck in the era for a decade, which alone gives this emu a huge advantage over other classic era emus.

=== Live EQ ===

Anybody serious about data collecting absolutely must level characters on the Live servers.  This is unavoidable.  If you're not willing to do this, then you are not going to be very helpful.  Luckily for us, the Test server offers people free gold account status, /testbuff for instant level 25, free lesson of the devoted, arena combat dummies to raise skills and double the experience gains; so leveling characters up isn't as time consuming as it sounds.  Mercenaries also speed up leveling tremendously.  AAs are now also granted for free. (yes really) Powerleveling is also very very fast if you can find somebody to do it for you.

The recommended classes to level are the following:

* High level Monk for parsing NPC hitpoints (don't get finishing blow AA) and FD for dumping aggro when determining NPC aggro radii.
* High level Shaman to parse NPC resists.
* High level bard to check for NPC immunities and just to get around (a lot of running to zones is involved).  Fade is also very useful and charm is required to parse some advanced statistics.  Bards can lockpick which is required for some zones.  A bard without finishing blow is also great for parsing NPC hitpoints.
* Level 65 Wizard with temperate flux staff to parse highly accurate NPC magic resist values.

Note that some other classes are required to parse some of the advanced statistics, but few (if any other than Torven) people will end up doing that.

The most effective powerlevling technique is to use a high level SK with a bard for the damage shield and use PBAoE +hate spells on the SK and cast PBAoE damage spells on the powerlevelee.  You can buy PBAoE potions from Zordak Ragefire in soldungb.

Suggested powerleveling method and locations: /testbuff to 25 then do kodiak caves outside velk's lab to 37 or so, then pull all the front spiders inside.  Pop Lesson of the Devoted right before pull dies.  Pull should get you to 50 or 51.  Immediately teleport out to PoK to preserve buff.  Head to PoNightmare and keep powerlevelee outside.  Pull everything in the front area from the werewolves to the zone-in, have powerlevelee come in and PBAoE, kill pull with DS, zone powerlevelee out to preserve Lesson buff.  Repeat process in PoValor with golem area mobs to level 65 or so.  Head to the back area in PoFire and camp out there to preserve the Lesson buff.  SKs can start swarming when the powerlevelee is around level 70.  Swarm in Fortress Mechanotus until the mobs turn blue if the SK is low enough to get exp there, then swarm Kaesora Library.

A note regarding AAs: It is recommended that your melee characters NOT get the 'finishing blow' AA.  Finishing blows will make parsing NPC hitpoints much more difficult and will at minimum decrease accuracy if not screw up the values entirely if you fail to avoid them.  Test server characters can however reset AAs to turn them all off at will using /resetaa.

As for the quality of Live server data, it's important to remember the following:

* NPC hitpoints and damages for common low to mid level NPCs were altered.  Damages should be taken from AK logs whenever possible.
* ALL PoP NPCs had their damages reduced by roughly 5% some time after Gates of Discord.
* Some PoP NPCs that were once slow immune are now slowable.
* With the exception of 'Mitigation of the Mighty', raid content was generally left alone, thankfully.
* The Test server has Mitigation of the Mighty on NPCs, but the production non-progression Daybreak servers do not.
* Sony globally reduced respawn timers some time ago.

Mitigation of the Mighty is a buff placed on raid NPCs to make them much stronger so progression servers would provide some modicum of challenge since Daybreak refuses to revert the many changes over the years that made the game much easier.  As of Oct 2016, raid content from Classic to Luclin has this buff.  ALWAYS mark down if the NPC you parsed had this buff.  Almost all of the NPCs that have this buff however were already parsed by Torven.  Parsing these NPCs on non-progression production servers is required to get accurate stats, so to get that information requires leveling production server (i.e. not Test) characters. (some stats are still obtainable with the buff however as explained below)

=== Logs ===

Log files are of course the most obvious source of data.  TAKP developers have a large collection of logs from AK which were submitted by users.  They were however submitted in confidence, so at a minimum some trust is required before logs are shared.  Old EQ logs from PC EQ are of course also usable.

Some things to keep in mind when examining logs:

* There was no shielding gear or Vie buffs on Al'Kabor, so AK logs are great for grabbing NPC damages.
* Shielding gear started to exist sometime around mid-2003.
* Vie buffs went Live with LDoN. (Sept 9 2003)
* Some NPCs have had their stats changed over time.  Favor AK logs.

If you are in possession of AK or old EQ log files but are hesitant to share them out of privacy concerns, please take a look at [[Chat_Text_Remover_Tool|this chat text remover tool]].

=== Patch Notes and Websites ===

Torven maintains a small archive of old saved web pages and files relevant to classic game mechanics and anything of research interest.

A large text file containing all known patch notes from 1999 through 2005 can be found [https://drive.google.com/open?id=0B9w2lZ-BZM3lUWJUZGVZVXFzT1E here]. (this file has notes that are missing in the http://www.tski.co.jp/baldio/patch/ archive)

The greater archive can be found [https://drive.google.com/open?id=0B9w2lZ-BZM3lUklnT3llZTA3OXc here].

Other links relevant to research:

[http://everquest.allakhazam.com/ Allakhazam]<br/>
[http://eq.magelo.com/ Magelo]<br/>
[https://archive.org/web/ The Wayback Machine]<br/>
[https://groups.google.com/forum/#!forum/alt.games.everquest alt.games.everquest Usenet Group]<br/>
[http://steelwarrior.xwarzone.com/forum/index.php The Steel Warrior Forum]<br/>
[http://www.eqclerics.org/ EQ Clerics Forum]<br/>
[http://www.thesafehouse.org/forums/ The Safehouse]<br/>

=== ShowEQ Data ===

A custom version of ShowEQ was used to collect NPC data and import it directly into the database.  Most of TAKP's NPC positions and other certain stats like body type are absolutely precise because of this.  This data is not yet publicly available.

ShowEQ also remains an important data source used on Live servers.  This wiki will not explain how to set it up as it is illegal to use on TAKP.  Serious new developers can contact Torven or Rob for that information.

The data most particular to ShowEQ is the following:

* NPC spawn locations
* NPC aggro and assist radii
* NPC respawn times
* NPC path routes
* Video of ShowEQ windows for scripted events

=== Torven's NPC Statistics Spreadsheet ===

Torven has a master spreadsheet with all of his NPC data and includes a long list of NPCs parsed on Al'Kabor.  This is noteworthy because many Live server common NPCs have had their hitpoints, damages, and in some cases attack delays altered.  The AK values must be used instead of Live's.

This file is not being made publicly available until some future date.  For now anybody wanting to collect data will need to contact Torven via PM on the forum to prevent duplicated work.  Read the last section of this page for a general idea of what TAKP needs.

== Tools and Methods ==

=== Spreadsheets ===

It is strongly recommended that would be data collectors use a spreadsheet application to store their data in.  [https://www.libreoffice.org/ LibreOffice] is a free office suite.

The reasons why this is desirable are:

* It's a convenient data store with organization, search and sorting features.  It allows you to easily see what needs to be worked on and what is done.
* It makes sharing data simple.  Just upload a file either directly to people or to a google drive which allows people to view the data with a link.  You could even share your data with multiple projects very easily this way.
* You have complete control over the data, so you know it's all from you and not edited.  Database entries can be and are changed by other developers.  Separate spreadsheet files hedge against data loss, error or corruption.
* It allows people to contribute without direct database access.  Entering values into the web editor is actually a small amount of the time that goes into content development.  Data collectors can merely fill out their own sheets and send them to TAKP staff who can enter the values into the database, which allows for greater security and lowers barriers to contributing.

[[File:stats_spreadsheet.png]]

Possible field names:

* NPC Name
* Level
* Zone
* Type (common, named, raid boss, raid trash)
* Era
* Hitpoints
* Min Hit
* Max Hit
* DB and DI if you want to get fancy.  Don't worry if you don't know what those are.
* Round size, or Innate Dual Wield (i.e. does this double or quad attack)
* Attack Delay
* Special Attacks (rampage, flurry)
* Spells (e.g. dragon breath weapons; innate procs.  Those could be two columns really)
* Slowability with slow mitigation (i.e. slow immune, not immune, or not immune but mitigated X%)
* Columns for the five resist values
* Aggro radius
* Combat regen
* Sees Invis?
* Sees Sneak? (very few NPCs have this)
* Summons?
* Immune to Charm?
* Immune to Mez?
* Immune to Lull?
* Immune to Snare/Root?
* Immune to Stuns?
* Aggro emote
* Death emote
* Slay emote
* A notes column for misc info

=== Text Editors ===

Discussing text editors may seem mundane, but there are a few things you should be aware of.

One, large text files break most text editors.  When dealing with very large text files, (such as logs) you need an editor that can handle them.  [https://www.emeditor.com/ EmEditor] was designed to handle large text files and has a free version.

Secondly you'll want an editor that can quickly count the number of text matches.  [https://notepad-plus-plus.org/ Notepad++] has this feature. (select text, hit ctrl-f then click count)  It also has block text selection (with alt+click and drag) and regex text matching and replace, which are handy features if you want to format text for pasting into spreadsheets.  Multi-line tabs at the top are also nice when you have a couple dozen files open.  Notepad++ also has a reliable autosave.  You only need this editor if you don't have large logs.

=== Grep ===

A good grep tool is absolutely required if you have logs to search.  [https://www.baremetalsoft.com/baregrep/ Baregrep] is recommended for Windows users.  This is even useful when merely playing and not developing.

Grep simply searches for for specified strings in one file or all files in a directory, then displays the filename(s) and lines containing the text.  Regex can be used to match multiple criteria.  E.g. '\S+' means match any string of 1 or more characters that aren't spaces; or put simply match any word.  Use a pipe (|) as an 'or' to match multiple strings.

Examples:

'''"\] Vulak`Aerr \S+ \S+ for"''' - display all of Vulak's hits on any target, regardless of weapon he may be wielding.<br/>
'''"assumes a defensive|lose the concentration to remain|YOU for "''' - display all lines with defensive discipline use, your discipline ending, and hits on you.

=== Video Recording Game Footage and ShowEQ Windows ===

Recording video is actually not difficult and can capture details that otherwise would not be preservable.  Recording scripted events or NPC path routes allows developers to recreate them with precision.  Furthermore a well-shot video can guarantee that events can be recreated with accuracy even if EQ Live were to shut down.

[http://www.fraps.com/ Fraps] is the old standard in video game recording and works well; however it's not free.  There are a number of alternatives available.  Try [https://obsproject.com/ Open Broadcaster Software] for a free program.

Recording video is generally straightforward and simple.  However often these programs leave you with a file that is both high quality and enormous.  This is actually a good thing, because you want more control over how the video is compressed.  After you shoot video footage, you'll want to save it in a compressed format.  [http://vidcoder.net/ Vidcoder] (free) is excellent for this task.  This will compress your video in H.264 which is a very good codec.  You can easily set the 'bitrate' which is how compressed it is.  You'll want to set it at a rate that uses the least disk space but still maintains enough detail so that developers don't lose important information from the video.  1,000kbit is a good bitrate for most situations.  You're not making a youtube video.

Tips for recording game footage:

* Record in half-size if your recording software supports it (Fraps does)
* Enlarge the default font size so that chat text is readable in half-sized video. (/chatfontsize)
* Recording in a lower frame rate may be worth the file size reduction.  (10 FPS should probably be the minimum)
* Utilize Feign Death characters to avoid interacting with scripts unnecessarily.
* Different F9 views may allow you to get more of the event in the shot.

To record ShowEQ windows, TAKP developers have been using [http://www.hyperionics.com/ HyperCam 2].  HyperCam 2 is free and allows you to select a region of the screen to record instead of a game window; in this way you can select only the portion of the ShowEQ window that you want to record.  A second monitor is ideal for this.  Compressing a large ShowEQ window video at 1,000kbit will result in something like 5 megs per minute video with very good clarity.

Tips for recording ShowEQ windows:

* Adjust the spawn list columns to display information useful for developers and exclude the useless columns.  i.e. show name, level, class, race, X, Y, Z, spawn time, run speed.
* Turn on NPC levels and names on the map.
* Turn off player following so the view area doesn't change when you move your character around.
* Record mob trails and turn them on for a portion of the video if the situation warrants.
* Sort the spawn list by respawn time so new spawns appear at the top for scripted events.
* Occasionally scroll the spawn list to show all the NPCs in the zone.
* Zoom in to show tightly bunched NPCs when appropriate.

=== Torven's Log Parsing Scripts ===

Torven has written a number of scripts that are used to pull data out of log files.  These scripts are required to determine certain NPC statistics.

The scripts are written in Lua, so you'll need a Lua interpreter to run them.  You can download a [http://luabinaries.sourceforge.net/download.html lua interpreter from sourceforge].

The [https://drive.google.com/drive/folders/0B9w2lZ-BZM3lMmxHMW9EQmsyMGs?resourcekey=0-eD4X4aE_vMYwW0b8Qpfnzw scripts are available on Torven's Google Drive].

To use them, put them into the same folder you extracted the Lua interpreter, open a command prompt, and execute the Lua binary with the script name as the only paramter.  i.e. 'lua53 attackspeed.lua'.  You will however need to edit the scripts before using them; they have variables that need to be set such as the log file name(s) and directory they are located in, and likely other variables that need to be set.

Some of these scripts produce a lot of output, so you'll likely want to redirect the output to a text file so the results pop up in your editor.  You can do this by appending a ' > filename.txt' at the command line.  If you use Notepad++ and the file you output to is already open in that editor, then Notepad++ will ask if you want to reload it after tabbing back to it and you can get the results in the editor.

[[file:lua_prompt.png]]

For non-programmers: it's important to understand how Lua comments work.  A double hyphen (--) means that the interpreter should ignore what is after that double hyphen; in this way you can 'turn off' code in the file, or write notes.  That's why you see multiple directories and file names.  Also be aware that directories need double backslashes or the script will not find the directory.

=== Other Log Parsers ===

To parse NPC hitpoints, you'll want to use a more traditional log parser.  [http://www.eq-companion.com/Downloads.html EQ Companion] is recommended, however it has some flaws: it cannot read DoT damage correctly and may give errors for the new damage shield messages on Live, so you'll want to kill stuff using only melee damage and direct damage spells.  It also is an old Visual Basic 6 app, so you may need to grab some extra files that it needs, which you can find on Torven's Google Drive [https://drive.google.com/open?id=0B9w2lZ-BZM3lRTZqclIxMDVINTQ here]. (you can also find them in old Win XP system folders or online DLL libraries)

The [http://gambosoft.eqresource.com/gamparsegettingstarted.php Gamparse] log parser also works.

== Basic NPC Statistics Collecting ==

=== Hitpoints ===

Parsing NPC hitpoints is straightforward.  Turn log on, kill mob rapidly with parsable damage, and try to get the final blow with a low damage hit.

A monk is preferable for this task as monks tend to cause damage with many smaller damage hits instead of fewer larger hits; plus they can use disciplines to increase attack speed.  Higher level AAs will increase the monk's minimum damage hit however.  You can refund all your AAs on Test by doing a /resetaa, or if you have free AAs, you can turn off auto-grant and relog.

Keep in mind:

* Killing rapidly is important to mitigate the effects of regen. (although most NPCs regen very slightly)
* Swap weapons to a low damage one before killing the NPC.
* Avoid casting damage shields as some parsers can't pick it up.  Third party damage shield hits were only very recently visible on Live.
* Use lower level characters if your higher level characters end up overkilling too much.  I.e. don't parse level 40 NPCs with a level 90 character.
* DO NOT GET the Finishing Blow AA.
* Avoid the crit damage AA as high damage crits may cause the final blow to be high damage.

=== Damage ===

Determining NPC damage is a bit more involved than parsing hitpoints.

The first thing to understand is that NPCs can only hit for 20 possible values.  This is advantageous to us because it allows us to predict an NPC's min or max hit from only a handful of data points.

Using Cazic Thule (in fearplane) as an example: his min hit is 220 and his max hit is 600.  This means he hits for 220, 240, 260...560, 580, 600.  You could easily look at some of his hits in a log to see that the damage interval is probably 20, then try to see which hit is likely the min or max and from there you can come up with a very good guess if not find the precise solution.

If you are producing Live logs, to get the max hit, you simply let the NPC hit you while sitting.  To get the min hit, you can increase your armor class to the point where the most common hit is the minimum.  You should verify that the min and max hit values produce the 20 value range that matches the hits you see before moving on to another NPC.

You can produce a 20 value distribution from a min and max hit in a spreadsheet by using following cell formulas:

A2 is the min hit<br/>
B2 is the max hit<br/>
C2 (this is the DB, or damage bonus value) =B2-(D2*20)<br/>
D2 (this is the DI, or damage interval value) =ROUND((B2-A2)/1.9, 0)/10<br/>
E2 to X2 (the 20 possible hits) =ROUND($C2+$D2*X, 0) (replace X with 1 through 20 for each column)<br/>

[[File:DBDI.png]]

Keep in mind:

* DO NOT WEAR SHIELDING GEAR.  This is CRITICAL.  Always check your character sheet for 0 shielding before getting min/max hits.
* DO NOT HAVE VIE BUFFS ON.  This is also critical.  Always check your buffs before getting NPC min/max hits.  Cleric mercs cast this!
* DO NOT USE A WARRIOR or knights with the Knight's Sedulity AA on.
* It's a good idea to collect a set of zero shielding gear and put that on before getting damages for some of the harder NPCs.
* Pets can be used to get NPC damage as well, but the higher level pets have shielding and are unusable.
* The damage interval is often not a whole number.

=== Attack Delay and Round Size ===

For attack delay, you'd need to use one of Torven's scripts.  Round size can be eyeballed, but the script will also give you this as well.  Attack delays are as important as NPC damage when tuning NPCs as they have a very large impact on the NPC's damage output.

The script used for this statistic is [https://drive.google.com/open?id=0B9w2lZ-BZM3lSW9oQTR4MzQ2X2s attackspeed.lua].  Edit it so it can find your log file.  The other default settings will suffice for most logs.

The script output looks like this:

<small><tt>
:Fight #6 [3] A fiery guardian - Fri Sep 16 19:23:26 2016 - Duration: 7m 34s<br/>
:Rounds: 81; Bashes: 16; Dmg Type(s): claw; Swings: 130 (On You: 130)<br/>
:rounds with 1 swings: 32 (39.5%)<br/>
:rounds with 2 swings: 49 (60.4%)<br/>
<br/>
:Avg pre-slow round delay: 1.583; Rounds/sec: 0.631; Swings/sec: 1<br/>
:Preslow Swings: 19;  Preslow time: 19;  Avg Swings/Round: 1.461<br/>
:Adjusting for 0 outliers, 0 double rounds, and 4 post-cast rounds<br/>
:Adjusted avg pre-slow round delay: 1.375; Adjusted rounds per second: 0.727<br/>
:(1)1(2)3(C1)1(2)1(2)2(2)1(C1)2(1)2(1)1(1)2(C1)1(2)2(C2)<br/>
:Avg post-slow round delay: 6.397; Rounds/sec: 0.156; Swings/sec: 0.255<br/>
:Postslow Swings: 111;  Postslow time: 435;  Avg Swings/Round: 1.632<br/>
:Adjusting for 0 outliers, 0 double rounds, and 55 post-cast rounds<br/>
:Adjusted avg post-slow round delay: 6.153; Adjusted rounds per second: 0.162<br/>
:6(1)7(C1)7(C2)6(1)7(C2)5(C2)7(C1)6(C2)7(C1)6(C1)6(C2)7(C2)7(C2)6(2)6(C2)6(C2)7(2)6(C2)7(C1)6(1)6<br/>
:(C2)7(C2)6(C1)7(C2)6(2)7(C2)6(C2)6(C1)7(C1)7(C2)5(C2)7(1)6(C2)7(C1)6(2)6(C1)7(C2)6(C2)7(C2)6(C2)<br/>
:6(C2)8(C1)5(2)7(C2)6(C2)6(C2)7(C1)6(C2)7(2)7(C2)5(C1)7(2)6(C1)7(C2)7(C1)5(1)7(C2)6(C1)7(C2)6(C2)<br/>
:6(C1)7(C2)6(C1)7(C2)6(2)6(C1)7(C2)6(C1)<br/>
<br/>
:Avg pre-slow bash delay: 8; Bashes per second: 0.125<br/>
:C8-C8<br/>
:Avg post-slow bash delay: 32.166; Bashes per second: 0.031<br/>
:Adjusted avg post-slow bash delay: 32; Bashes per second: 0.031<br/>
:-C32-C32-C32-C32-C32-C32-C33-C32-C32-C33-C32-C32-C32<br/></tt></small>

You can easily see what the round size is (doubles in this case, obviously).

The 'chain' of numbers are the round sizes in swings and seconds between rounds.  The numbers in parentheses are the number of swings (hits and misses) for the melee round.  Bashes just show the time in seconds between kicks/bashes.  The Cs indicate that the NPC casted a spell some time before that round or bash.  This is noteworthy because spellcasts make the estimates less accurate.

You can see that the script produces two estimates for attack delay: 1.583 and 1.375 unslowed, plus 6.397 and 6.153 unslowed.  The 'adjusting' is from ignoring rounds that followed a spell cast, which is useful when the spell casts have cast times. (in this case, they don't)  The slow was Turgur's, which is a 75% slow and which multiplies attack delay by 4.  The attack delay for this NPC is most likely 1.6.  Often the 'adjusted' estimate is actually slightly lower than the actual value.

Keep in mind:

* Parse for several minutes at least and duck and/or turn your back to avoid ripostes on melee characters. (or hold a fire beetle eye).  The script will try to remove ripostes, but not all ripostes give the message.
* If the NPC is full slowable, then parse it with Turgur's on, then divide the slowed delay estimate by 4 to get a more accurate value.
* Most, and possibly all, NPCs have a delay in the tenths of seconds.
* Bashes are always 8 second apart.  If they are less than 8 seconds apart, then the NPC is hasted by some spell and should be dispelled and reparsed.  (note that Rampage and Flurry reset the bash timer)
* Spell casting NPCs will need longer logs, as spell casts mess with their attack delay timer.  Let NPCs OOM first if they are limited by mana.
* This script will also adjust for 'outlier' rounds, which are rounds that have a much larger delay than the average.
* NPCs with very low attack delays will end up with multiple rounds in the same log second, which will make the parser think that they are just one big round and result in a slower than actual attack delay estimates.  For fast hitting NPCs, always Turgur it and divide the slowed delay by 4.  Luclin raid NPCs are known to hit fast.

=== Slow Mitigation ===

NPCs starting in Planes of Power often will mitigate slow spells.

Determining this value is a bit tricky.  You need to parse both the slowed and unslowed delay with as much precision as possible, then do some math.

Using three input variables: Unslowed Delay, Slowed Delay, and Full Slow Strength (i.e. 75% for Turgur's)<br/>
Actual Slow Strength = 1 - Unslowed Delay / Slowed Delay<br/>
Slow Spell's Effectiveness = Actual Slow Strength / Full Slow Strength (this is the number in Torven's spreadsheet)<br/>
NPC's Slow Mitigation = 1 - Slow Effectiveness (this is the number the database uses, as a integer)<br/>

The easiest way to do this is to use a spreadsheet.  Here are the cells I use:

C13 = Unslowed Delay, in seconds<br/>
C14 = Slowed Delay, in seconds<br/>
C15 = Slow % (e.g. 75 for Turgur's)<br/>

C18 =1-(C13/C14)<br/>
C19 =C18/C15<br/>

Note: C15, C18 and C19 should be percent formatted.

[[File:slowmitcalc.png]]

The database uses 0 to mean no mitigation, so the higher the value, the more the mitigation.

The mitigation ranges corresponding to the client slowed messages are the following:

* 80-99%: Slightly
* 35-80%: Partially
* 1-30% Mostly

The best way to determine the mitigation is to use the slowed bash delay, because unslowed bash delay is always 8 seconds.  The longer delay on bashes reduces the margin of error.  Not all NPCs bash however, and Flurries and Rampages will reset the bash timer, so you'll have to eyeball what the real bash delay is for special attacking NPCs or use the attack round delay instead.

Note that Sony almost always made the mitigation value a multiple of 5.

=== Special Attacks ===

Special attacks are Rampage, Flurry, and Wild Rampage.  These are also parsed with [https://drive.google.com/open?id=0B9w2lZ-BZM3lSW9oQTR4MzQ2X2s attackspeed.lua].

The script will give you two numbers: one for specials per round, and one per hit.  The database wants the per round number.

Low margins of error require long logs.  For raid bosses, you would want to tank them for at least an hour.

=== Contact/Innate Procs ===

Contact procs are spells that NPCs may have that randomly hit their target if they land a melee blow.  These don't function like weapon procs-- the chance they have to trigger is not scaled by attack delay and they only trigger on swings that hit.

The chance an NPC may trigger a contact proc per hit varies significantly depending on the NPC and has a large effect on the difficulty, so this should not be ignored or guessed at.  It is however simple to figure out.

First you generate a log with as many hits from the NPC as possible; keep sitting down so the NPC hits you more often.

Next you open your log in Notepad++ or use a grep tool to count the number of times the spell hit you.  Ensure you check for both full resists and unresisted spell hits and add them.  Make sure you're not accidentally counting the spell from other NPCs than the one you're checking; split the log if you need to.

Then you add all the NPC's hits on you.  This includes kicks and bashes, which can also proc innates.

The proc chance is merely: procs / hits.  Note that you really want something like 300+ hits at the minimum to get a reasonable estimate.

One of Torven's scripts, [https://drive.google.com/open?id=0B9w2lZ-BZM3lUGlybDdxUEZxUnc innate_procs.lua], will parse logs for contact procs and output the proc rate if you input the NPC name, spell name and spell resist message if you would prefer to do it that way.

=== Aggro Radius ===

Determining aggro radius is very simple.  Load ShowEQ, target the mob, duck your character to move slow, and inch up to it while keeping an eye on your distance to it.

It's important to keep these in mind however:

* NPCs don't react immediately on Live.  If you do it too fast, you'll end up with radius estimates that are too small.  Inch slowly.
* NPC radii are virtually always (if not always) multiples of five.  If you register a value not divisible by five, you inched too fast.
* Since they are divisible by five, you can test by stopping one unit above, pausing, inch to one unit below, and pausing for each multiple of five.

Assist radii are tricky to figure out.  There are a number of techniques.

If the assist radius is larger than the aggro radius or you are not KoS, then you can use a couple of FDing or fading characters, then move two NPCs some distance apart, taking note of the distance, then and FDing one of them and seeing if it assists.

Another trick would be to snare two mobs to park them, place them some distance apart, feign, then pull one from a distance and see if the other assists.

If they don't see invisible and some of the NPCs are close together somewhere, you can use invis to stand on one of them to get the distance to the other(s) in ShowEQ, then pull from a distance.

=== Immunities ===

These are very straightforward, however keep the following in mind:

* If you use a stun spell to check stun immunity, be aware of the spell's level cap as most stuns don't work on level 56+ NPCs.
* Monks have an AA called Stunning Kick that you can use to check for stun immunity.
* Monks have an AA called Crippling Strike that you can use to check for snare immunity.
* Monks get lulls.
* Bards can check for all immunities except for stun immunity on level 56+ NPCs.

=== Resists ===

Resists require more time and effort than the other basic statistics.  The basic procedure is to cast hundreds of spells on an NPC then run a script on the log to spit out a resist estimate.  You will of course need to cast spells from all five resist types.  To do this you will want to run an automated tool to do the spell casting for you.  The more spells you cast, the smaller the margin of error.  Generally you'd want to spend somewhere around 2 hours per NPC or more if accuracy is a concern.

The best class to do this on is a shaman, as they can cast spells for all five resist types.  Wizards however are the best to measure magic resist because they can use a staff of temperate flux which is an instant-click item that casts a weak debuff.

Torven wrote an [https://www.autoitscript.com/site/autoit/ Auto-IT] script to automate spell casts, which is called [https://drive.google.com/open?id=0B9w2lZ-BZM3lZjFBa0x2N0pWWVE 2spam.au3].

This script assumes that: (you can edit the script to change these if you like)

* The spell you want to automate is casted by key 2.
* Turgur's Swarm (the AA ability) is casted by key 7.
* A malo debuff is casted by key 4.

The script is controlled by the numpad.  Make sure numlock is on.  You can change the delay between casts by hitting numpad plus or minus.  The maximum delay is 10 seconds.  Hitting numpad 2 through 7 will start or stop the casting.

These are the casting options:

* Numpad 2 will hit key 2 every X seconds.
* Numpad 3 will hit key 2 every X seconds and also hit Turgur's swarm (key 7) every cooldown.
* Numpad 4 will hit key 2 every X seconds and also hit Turgur's swarm (key 7) every 60 seconds.
* Numpad 5 will hit key 2 every X seconds and also hit Turgur's swarm (key 7) and Malo (key 4) every 120 seconds.
* Numpad 6 will hit Malo (key 4) every 120 seconds.
* Numpad 7 will hit key 2 every X seconds and also Malo (key 4) every 120 seconds.

On a shaman, the spells you would typically want to use are Burst of Flame, Frost Rift, Disempower, Tainted Breath and Sicken; however a higher level cold DD and poison DD is desirable in some circumstances. (explained later)

If parsing lower level NPCs, you will probably need to charm it and cast a regen spell on it so you don't kill it.

It's very important to understand how level difference affects resist rate.  You need to input your level and the NPC's level into the log parsing script before you run it, otherwise the estimates will be wrong.  The script has no way to know if you failed to do this.  Make very certain that the levels are set correctly.

The level differential resist modifier works like this:

* If the NPC is level 67 or higher and the player is level 65 or higher, then there is no resist modifier.
* If the NPC is level 66 or lower, then the resist modifier the player gets is INT(levelDiff * levelDiff / 2) capped at 40.
* So the simple thing to remember is, when parsing level 66 and under NPCs with high level (9+ levels above) characters, your spells will hit them as if they had a -40 debuff on them.

The script used to parse NPC resists is [https://drive.google.com/open?id=0B9w2lZ-BZM3lMmlRSHYzTEdDYTA bossresists.lua]. (although it should be renamed as it works on any NPC and is used on common NPCs)

For this script to work, you need to remove spell focus items from your character so that the spells do unmodified damage.  Focus items will make your spells do random amounts of damage, and this needs consistant spell damage to work.

You also need to edit the script and either add the spells you intend to cast, or change the damages for the spells already there.  Use double hyphens to comment out spells you aren't casting.  This script will not function/output bad data if the spell damages are incorrect.

The output looks like this:

<small><tt>
:Resist Modifier From Level Difference for All-Or-Nothing spells == 0
:Resist Modifier From Level Difference for Direct Damage spells == 0
:Ensure that levels are set correctly - PC: 93  NPC: 70
:--------------------------------------------------------------------------
:Neffiken Lord of Kelek`Vor resists.txt
:Disempower - 309 casts; margin of error: 5.5%
:Full Hits: 114 (36.8%)  Full Resists: 195 (63.1%)
::   Est Resist Value Based on Full Resists: 127 +/-12
<br/>
:Malosinete II - No casts
:Turgur's Swarm II - No casts
:Burst of Flame - 350 casts (138 critical); margin of error: 5.2%
:Full Hits: 232 (66.2%)  Hits: 306 (87.4%)  Full Resists: 44 (12.5%)
::   Est Resist Value Based on Full Hits: 68 +/-11
::   Est Resist Value Based on Full Resists: 76 +/-32
<br/>
:Frost Rift - 144 casts (13 critical); margin of error: 8.1%
:Full Hits: 0 (0%)  Hits: 27 (18.7%)  Full Resists: 117 (81.2%)
::   Est Resist Value Based on Full Hits: >479
::   Est Resist Value Based on Full Resists: 488 +/-49
::   Est Resist Value Based on Lowest Non-Crit Partial Mod: 503 +/-21
::   Est Resist Value Based on Lowest Crit Partial Mod: 494
:Ice Strike - No casts
:Tainted Breath - 25 casts; margin of error: 19.6%
:Full Hits: 0 (0%)  Full Resists: 25 (100%)
::   Est Resist Value Based on Full Resists: >180
<br/>
:Shock of the Tainted - 183 casts (19 critical); margin of error: 7.2%
:Full Hits: 0 (0%)  Hits: 39 (21.3%)  Full Resists: 144 (78.6%)
::   Est Resist Value Based on Full Hits: >465
::   Est Resist Value Based on Full Resists: 473 +/-44
::   Est Resist Value Based on Lowest Non-Crit Partial Mod: 464 +/-3
::   Est Resist Value Based on Lowest Crit Partial Mod: 466
:Sicken - 227 casts; margin of error: 6.5%
:Full Hits: 158 (69.6%)  Full Resists: 69 (30.3%)
::   Est Resist Value Based on Full Resists: 61 +/-14
<br/>
:Putrid Decay - No casts</tt></small>

The margin of error confidence interval is 95%, meaning there is a 19 in 20 chance that the resist value falls within the given range.

Note that it's important to only have one NPC per log file, as full resists do not give the target name, so it's impossible for the script to determine which NPC the spell was cast on.

The script has several ways to compute estimates, and depending on the resist value of the NPC, some methods become available and others unavailable.

If the NPC's effective resist value (after the resist mod from level difference) is 0-199, then all-or-nothing spells can provide an estimate.  If the value is 200 or over, then they will all fully resist.  This is also the case for full damage direct damage hits.

Direct damage full resists can provide an estimate from 0-599, as direct damage spells become immune at 600 resist instead of 200.  Estimates based on full resists are 3 times less accurate however.

If the NPC's resist value is 200+, then direct damage partial hit damage can be used to compute an estimate.  The way this works is that the maximum damage a direct damage spell can hit for is gradually reduced down to zero from 200 to 600 resist, so if you cast many spells you can take the highest damage and calculate an estimate from that.  A higher damage spell must be used however. (something close to 600 damage is preferable but over that doesn't help)  For this reason it's recommended to use a higher damage DD spell on NPCs with greater than 200 resist.  The magin of error range for this estimate isn't precise however.

If the NPC has >600 resist, then you'll either need to use a wizard lure spell then add 300 to the estimate, or put a debuff on it and carefully ensure that all spell casts landed with that debuff on then add the debuff's resist reduction to the estimate.  If however the NPC fully resists like a dozen casts from a lure, then it's likely that the resist value is 1000+ so you can probably move on.

Wizards can use a temperate flux staff to parse extremely accurate magic resist estimates.  If the NPC has >200 MR, then debuff it so that the staff will hit.

Low level NPCs will probably have resists 40 or under, so parsing these with high level characters will only result in estimate readings of <40somthing.  Fortunately there are resist buffs that grant precisely 40 resist value, so you can charm the NPC and cast these buffs on it, then parse it and either lie to the script and raise the level to an even con or subtract 40 from the displayed estimates.  Take great care not to let the buffs drop.

=== Combat Regen ===

This is a low priority stat, but if you're a completionist, here's how to get it.

You'll need to use some video recording software for this.  Damage the NPC a good amount, then sit there and record the game as the NPC attacks you.  You'll have to do something to keep from losing aggro every 10 minutes.  Also, you can record the game at a low frame rate to reduce file size.

Depending on how slow the NPC regens, at the minimum you'll want to capture the NPC regenerating two percentage points, and preferably more.  This is time consuming because most NPCs regen very little.

Load the video in a player, seek through the video, jot down when the HP went up and calculate how many ticks were between.  At this point you'll need to know the NPC's max HP, which will require killing it.  Ensure that the NPC heals to full health before doing so.

Once you know how much HP is 1% for the NPC, it's a simple matter to divide that number by the number of ticks (1 tick = 6 seconds) it took the NPC to regen 1%.  It's more accurate to allow it to heal more than 1% then figure out how many ticks were between start% and end% and divide that by end% - start%.

=== Spawn Chance and Drop Rates ===

There is a simple method to estimate rare dungeon NPC spawn chances, and also loot drop rates: buy a merc and sit on the spawn with a log running; also be grouped (the rest of the group does not need to stay online) with the advanced looting system on so that 'no one was interested in the items' text is logged.  You can then check the log for the number of spawns, placeholders, and loot.

If the NPC is non-KoS, you could set up a script/bot (using autoit using a script linked above would work) to cast PBAoE spells periodically.

Generally you would want to do this for at least 48 hours, depending on the spawn rate.


== Advanced NPC Statistics ==

Advanced statistics include NPC offense, mitigation, avoidance and to-hit values.  Calculating these values requires much more effort than do the basic statistics.  Due to the length, these will have their own pages.

For NPC offense, a level 65 warrior with gear will be required to tank and it will require one or two automated healers. (mercs don't work for the harder NPCs)  Since each automated character requires a dedicated machine, this could require up to 3 physical machines.  Furthermore, to-hit cannot be obtained from Mitigation of the Mighty'ed NPCs, so that has to be done on the production servers and not Test.

[[Calculating_NPC_Offense_and_To-Hit|Calculating NPC Offense and To-Hit]]

NPC mitigation can be obtained from using a high level tank and a lower level character to attack the NPC, however it's generally more desirable to parse this at the same time as offense/to-hit so all four stats can be obtained from the same log/session if you are able to do so.

[[Calculating_NPC_ Mitigation_and_Avoidance|Calculating NPC Mitigation and Avoidance]]

== The Data TAKP Still Needs ==

The vast majority of raid bosses are already parsed up through and including PoP.  Primarily what is needed are the more common NPCs in Luclin and PoP.  PoP NPCs in particular have more to them in the form of slow mitigation, resists and immunities and it's more important to get those right since it's the end expansion.

Serious contributers can contact Torven via PM on the forum to get a quick answer as to what to go after.