Analyzing multiple CSV counter log files could be fined tuned

Feb 3, 2009 at 1:45 AM
I found out you you can analyze multiple counter logs today. However, PAL issues a note that it can only do this effectively for binary (*.blg) log files. So it attempts to (I think) convert the CSV files to binary and merge them for analysis. However, it fails with the error below.

But I was able to generate a report by manually relogging the logs (individually) from CSV to binary format and then supply the list of binary logs to PAL to parse and that worked out fine. Luckily, my logs were small so it was quick to change formats. But it looks like the multiple logs, merging support could be fine tuned for non-binary formatted logs. Don't understand why my manual relogging worked out fine while PAL's attempted conversion failed. I didn't look through the source code to see what was wrong.

C:\vptools>CScript "C:\Program Files\PAL\PAL v1.3.4.2\PAL.vbs" /LOG:"C:\vptools\
vpmem.csv;C:\vptools\vpcpu.csv;C:\vptools\vpheap.csv" /THRESHOLDFILE:"C:\Program
 Files\PAL\PAL v1.3.4.2\VerticalViewPointPerfLogAnalysis4LoadTestsMultipleVpOn1P
c.xml" /INTERVAL:"AUTO" /ISOUTPUTHTML:True /ISOUTPUTXML:False /OUTPUTDIR:"C:\tmp
" /HTMLOUTPUTFILENAME:"[LogFileName]_PAL_ANALYSIS_[DateTimeStamp]_[GUID].htm" /N
umberOfProcessors:1 /ThreeGBSwitch:False /SixtyFourBit:False /TotalMemory:1 /Ker
nelDumpOnCDrive:True
Microsoft (R) Windows Script Host Version 5.6
Copyright (C) Microsoft Corporation 1996-2001. All rights reserved.

PAL.vbs v1.3.4.2

Merging perfmon logs together. This can be a time consuming process...
Executing: ReLog.exe C:\vptools\vpmem.csv C:\vptools\vpcpu.csv C:\vptools\vpheap
.csv -f BIN -o C:\DOCUME~1\tvqa620\LOCALS~1\Temp\{4D1C7881-6BBD-43F7-B21E-D24F96
9B54C5}\_MergedPerfmonLog.blg

Input
----------------
File(s):
     C:\vptools\vpmem.csv (CSV)
     C:\vptools\vpcpu.csv (CSV)
     C:\vptools\vpheap.csv (CSV)

Error: To relog multiple files all files must be in binary format.

Waiting for C:\DOCUME~1\tvqa620\LOCALS~1\Temp\{4D1C7881-6BBD-43F7-B21E-D24F969B5
4C5}\_MergedPerfmonLog.blg to be created...
C:\Program Files\PAL\PAL v1.3.4.2\PAL.vbs(342, 5) Microsoft VBScript runtime err
or: File not found

C:\vptools>
Feb 3, 2009 at 1:50 AM
On further thought. The failure to create a merged binary log by relog might be due to each log having slightly different sample timestamps as they weren't synchronized or started at the same time. But PAL had no problems parsing all the logs together when I relogged them to binary format individually (not merged) and supplying the individual list of logs to PAL.
Coordinator
Feb 4, 2009 at 2:23 AM
Right, only binary logs can be merged together by relog. PAL doesn't use relog to covert CSV files to binary - I just never added that as a feature. Pleaes keep in mind that it is highly recommended to only capture logs in binary (BLG) format. Perfmong logs that record in CSV format fail to track new instances of processes - meaning if you start the log, then a new process starts, then perfmon fails to track the new process. It trackes all new process instances fine when capturing in BLG. You should only record perfmon logs using binary format. With that in mind, i likely won't add the conversion of CSVs to BLGs unless there is a high demand for this.
Feb 20, 2009 at 4:24 PM
Hi guys,

First of all; great piece of software this !

Second; it doesn't like me...

All my files are in binary format, yet still it refuses to merge them. Is there something i'm doing wrong ? The files are all from one logging session, no timestamp discrepancies or anything like that. I'm lost for words...

What do i need to do to merge them ? The native OS command relog.exe throws the same error, by the way...

C:\>CScript "C:\Program Files\PAL\PAL v1.3.4.3\PAL.vbs" /LOG:"C:\SQL_Server_Perf
ormance_Trace_000020.blg;C:\SQL_Server_Performance_Trace_000008.blg;C:\SQL_Serve
r_Performance_Trace_000009.blg;C:\SQL_Server_Performance_Trace_000010.blg;C:\SQL
_Server_Performance_Trace_000011.blg;C:\SQL_Server_Performance_Trace_000012.blg;
C:\SQL_Server_Performance_Trace_000013.blg;C:\SQL_Server_Performance_Trace_00001
4.blg;C:\SQL_Server_Performance_Trace_000015.blg;C:\SQL_Server_Performance_Trace
_000016.blg;C:\SQL_Server_Performance_Trace_000017.blg;C:\SQL_Server_Performance
_Trace_000018.blg;C:\SQL_Server_Performance_Trace_000019.blg" /THRESHOLDFILE:"C:
\Program Files\PAL\PAL v1.3.4.3\SQLServer.xml" /INTERVAL:"AUTO" /ISOUTPUTHTML:Tr
ue /ISOUTPUTXML:False /HTMLOUTPUTFILENAME:"[LogFileName]_PAL_ANALYSIS_[DateTimeS
tamp]_[GUID].htm" /NumberOfProcessors:4 /ThreeGBSwitch:False /SixtyFourBit:False
 /TotalMemory:4 /KernelDumpOnCDrive:True
Microsoft (R) Windows Script Host Version 5.7
Copyright (C) Microsoft Corporation. All rights reserved.

PAL.vbs v1.3.4.3

Merging perfmon logs together. This can be a time consuming process...
Executing: ReLog.exe C:\SQL_Server_Performance_Trace_000020.blg C:\SQL_Server_Pe
rformance_Trace_000008.blg C:\SQL_Server_Performance_Trace_000009.blg C:\SQL_Ser
ver_Performance_Trace_000010.blg C:\SQL_Server_Performance_Trace_000011.blg C:\S
QL_Server_Performance_Trace_000012.blg C:\SQL_Server_Performance_Trace_000013.bl
g C:\SQL_Server_Performance_Trace_000014.blg C:\SQL_Server_Performance_Trace_000
015.blg C:\SQL_Server_Performance_Trace_000016.blg C:\SQL_Server_Performance_Tra
ce_000017.blg C:\SQL_Server_Performance_Trace_000018.blg C:\SQL_Server_Performan
ce_Trace_000019.blg -f BIN -o C:\DOCUME~1\A14690~1.EUR\LOCALS~1\Temp\{7226623E-D
182-4AC7-BCF1-E05C2CC8275C}\_MergedPerfmonLog.blg

Input
----------------
File(s):
     C:\SQL_Server_Performance_Trace_000020.blg (Windows 2000)
     C:\SQL_Server_Performance_Trace_000008.blg (Windows 2000)
     C:\SQL_Server_Performance_Trace_000009.blg (Windows 2000)
     C:\SQL_Server_Performance_Trace_000010.blg (Windows 2000)
     C:\SQL_Server_Performance_Trace_000011.blg (Windows 2000)
     C:\SQL_Server_Performance_Trace_000012.blg (Windows 2000)
     C:\SQL_Server_Performance_Trace_000013.blg (Windows 2000)
     C:\SQL_Server_Performance_Trace_000014.blg (Windows 2000)
     C:\SQL_Server_Performance_Trace_000015.blg (Windows 2000)
     C:\SQL_Server_Performance_Trace_000016.blg (Windows 2000)
     C:\SQL_Server_Performance_Trace_000017.blg (Windows 2000)
     C:\SQL_Server_Performance_Trace_000018.blg (Windows 2000)
     C:\SQL_Server_Performance_Trace_000019.blg (Windows 2000)

Error: To relog multiple files all files must be in binary format.

Waiting for C:\DOCUME~1\A14690~1.EUR\LOCALS~1\Temp\{7226623E-D182-4AC7-BCF1-E05C
2CC8275C}\_MergedPerfmonLog.blg to be created...
C:\Program Files\PAL\PAL v1.3.4.3\PAL.vbs(342, 5) Microsoft VBScript runtime err
or: File not found



Thanks in advance !

Feb 20, 2009 at 4:42 PM
Got it... Win2K files aren't friends od XP technology... Aww that sucks :-(
Feb 20, 2009 at 9:05 PM
Hmm...that's too bad. Maybe you can try using the Win2k version of relog (on a Win2k PC)?

I haven't tried relog on Win2k PC and I haven't tried relog'ing Win2k perflogs, but I can tell you this, typeperf.exe on WinXP/2k3 works against logging a remote Win2k PC. So the other command line perfmon tools like logman.exe may work against a remote Win2k PC as well.

http://www.microsoft.com/downloads/details.aspx?FamilyID=F043C2F5-2A48-41ED-951B-BA7F62CF51D6&displaylang=en
Oct 9, 2009 at 1:21 AM

FYI, for anyone using multiple CSV perflogs like me who would like to use them with PAL, here's how I preprocess/merge them before using PAL:

relog perflog1.csv -f bin -o perflog1.blg
relog perflog2.csv -f bin -o perflog2.blg
...and so on for additional logs
relog perflog1.blg perflog2.blg [more as needed] -f bin -o mergedPerflog.blg
del perflog1.blg perflog2.blg [more as needed]

Then use mergedPerflog.blg with PAL

Also, the reason, I use multiple CSV perflogs, is because I create the CSV perflogs with typeperf.exe and text based counter list file that I supply to typeperf. Easier to setup on demand & reconfigure and faster to use than perfmon.

And also, because I'm monitoring the same application across many machines, I separate the perfmon logging to one log for CPU data across all machines, one log for memory data across all machines, etc. So then I finally merge them all into one log for PAL when I need to do the PAL analysis. And yet, for standalone raw data analysis, I can simply view the CSV files with ease, and view by group (CPU use only, memory use only, etc.)

Coordinator
Oct 12, 2009 at 4:27 PM

In PAL v2.0, I have placed all of the _Total instances on their own chart. This should help with your merged perfmon logs.

Apr 9, 2010 at 5:53 PM
Edited Apr 9, 2010 at 6:12 PM

Hi
your software rocks !

As I am still supporting w2K boxes, I was looking for details on the following error (Error: To relog multiple files all files must be in binary format.), that was occuring when trying to merge daily blg files to get a weekly report
google pointed my this thread.

C:\Temp\perflogs\test_lea>ReLog.exe MS_IIS_5_6_NTINTFR00046_20100405.blg MS_IIS_5_6_NTINTFR00046_20100329.blg MS_IIS_5_6_NTINTFR00046_20100330.blg MS_IIS_5_6_NTINTFR00046_20100331.blg MS_IIS_5_6_NTINTFR00046_20100401.blg MS_IIS_5_6_NTINTFR00046_20100402.blg MS_IIS_5_6_NTINTFR00046_20100403.blg MS_IIS_5_6_NTINTFR00046_20100404.blg -f BIN -o _MergedPerfmonLog.blg

Input
----------------
File(s):
     MS_IIS_5_6_NTINTFR00046_20100329.blg (Windows 2000)
     MS_IIS_5_6_NTINTFR00046_20100330.blg (Windows 2000)
     MS_IIS_5_6_NTINTFR00046_20100331.blg (Windows 2000)
     MS_IIS_5_6_NTINTFR00046_20100401.blg (Windows 2000)
     MS_IIS_5_6_NTINTFR00046_20100402.blg (Windows 2000)
     MS_IIS_5_6_NTINTFR00046_20100403.blg (Windows 2000)
     MS_IIS_5_6_NTINTFR00046_20100404.blg (Windows 2000)
     MS_IIS_5_6_NTINTFR00046_20100405.blg (Windows 2000)

Error: To relog multiple files all files must be in binary format.

I found a simple workaround : relog the single files first from (Windows 2000) from to (Binary) : with something like
relog MS_IIS_5_6_NTINTFR00046_20100405.blg -f bin -o reloged\MS_IIS_5_6_NTINTFR00046_20100405.blg

and then PAL will work with merged W2000 blg files
Vincent