Retrieving variables is incredibly slow

megajar wrote on Fri Aug  4 17:14:26 MEST 2006:

The previous problem with the Java/Perl project has been solved, but now
I have a new problem. At a certain point in the Perl code, it uses the retrieve()
function to fetch tabular data from a file, and stores it in a hash variable.
It does this several times in a row, retrieving data from several different
files -- and these files are very big. Once all the data is loaded in, it's
used to perform the necessary processing.

The problem is, when I'm stepping-through the code line by line, everything
works fine up until I get to the point where the retrieve() functions start.
After the first retrieve(), the debugger's "Retrieving Variables" process
takes much longer than before. After the second retrieve(), it takes even
longer. And by the time all the retrieve() functions are done, the "Retrieving
Variables" process has grown to an excruciatingly long wait time. And now,
this long wait must be repeated after EVERY LINE after this! Even a simple
print statement causes the variables to be reloaded, which now takes forever!

I can't figure out how to fix this. I know the long wait is caused by the
large volume of data being loaded into memory. I don't even need to view
that data. But I can't turn it off! I tried using the Variables View menu
to disable displaying local variables, but that didn't help. I tried disabling
global variables, but no luck. I even tried disabling displaying ALL variables,
but it STILL takes forever.

Can anyone help me fix this problem? I finally managed to figure out how
to make the Perl and Java interact and still be able to step-through the
Perl; but unless I can get rid of this horrendous load-time after EVERY
STATEMENT, solving that first problem will be effectively pointless.

jgangemi wrote on Fri Aug  4 18:58:05 MEST 2006:

this sounds like a potential bug in the debugger, but unfortunately, there
isn't going to be a quick fix for this.

i'm currently working on cleaning up some of the debugger code as i have
noticed other bugs (such as output not being printed when stepping over
subroutines containing print statements), but i'm still a long way off from
having anything ready to commit. 

would it be possible for you to just use a smaller subset of data for now
in your testing? how about testing both pieces of the system in isolation
using mock objects and/or stubs? 

megajar wrote on Fri Aug  4 19:14:49 MEST 2006:

Unfortunately, no can do. The Perl program's function is to validate text
messages against a given messaging standard. The information about the standard
itself (syntax rules, etc) is contained in those files that get loaded in.
Without them, the program can't do anything.

Thanks anyway :-(

jgangemi wrote on Fri Aug  4 19:20:23 MEST 2006:

but what exactly are you trying to test? 

you should be able to create some canned data that matches valid/invalid
that you could use for basic unit testing.

the fact that you are unable to decouple the two for testing purposes throws
up lots of red flags in my mind.

megajar wrote on Fri Aug  4 22:19:58 MEST 2006:

It's not that the two pieces can't be decoupled, it's that I don't know
how to do it.

The files containing the standard are not created by us; they're supplied
to us from an outside company, who releases a new version annually. The
standard is made up of a series of messages; each of those contain a series
of large data structures; and each of THOSE contain a series of smaller
data structures. It's a complicated hierarchy, and all the identifiers and
tables have to match up correctly.

Reproducing a scaled-down version of this hierarchy would be difficult enough
as it is; but I'm relatively new to the project, and I don't understand
the full layout of the hierarchy well enough to reproduce it.

Thanks anyway, but for now, I'm going to try a different Perl IDE. EPIC
is really convenient to use, but until this bug is fixed, it won't be able
to meet my needs.

Thanks anyway for the help, though. :-)

mtwomey wrote on Thu Sep 28 19:57:54 MEST 2006:

Yes, I am experiencing this exact same issue when reading a large file into
an array. When I do an strace on the perl process, I see groups of writes
burst out (10 or 20) then a long (10 second or so) pause and then another
burst.

I'm not sure what's causing this because when I debug it via command line
this doesn't happen. It literally takes 10 minutes or so for every step
after this array is loaded.

Can you tell me what you are using as an alternative?

Thanks,

-Matt

jploski wrote on Sun Oct  1 15:05:52 MEST 2006:

The EPIC part of the debugger uses regular expressions to find out whether
perl -d's output contains a command or session termination token. It used
to apply the regex to the whole accumulated output, over and over again.

Try 0.4.13 or 0.5.11, it should be much snappier.

jploski wrote on Fri Aug  4 20:22:38 MEST 2006:

I have also seen this happen before. There is no known workaround. I posted
a bug report:

https://sourceforge.net/tracker/index.php?func=detail&aid=1534713&group_id=75859&atid=545274

megajar wrote on Fri Aug  4 22:20:34 MEST 2006:

Thanks Jan.